Troubleshooting

Start at the earliest failing stage. Record build identity and incident time, and retain the WAL. Repeated restarts are not a substitute for locating the failure.

No matching target

Match match.executable exactly to /proc/PID/exe. Check UID, cgroup, Pod labels, and namespace. Look for equal-priority conflicts, exclusions, max_targets, and overlap between managers. A short-lived process can exit before polling or attachment.

sudo netflowscout status --component endpoint \
  --config /etc/netflowscout/netflowscout.json \
  --endpoint-config /etc/netflowscout/endpoints/billing.json

Target found, attachment fails

Run doctor. Check BTF, root/effective capabilities, target PID/filesystem visibility, libssl mappings, Go version, and matched probe/BPF objects. HTTP/3 also requires DWARF, symbols, and supported quic-go. Set an explicit module for mixed runtimes. Broader protocol signatures cannot fix an unsupported runtime layout.

Attached, but no business observations

Generate new requests after attachment. Check application_source, protocol/direction/port detection, stream socket wiring, and matching plugin directories. Inspect capture, bridge, and analyzer drops, incomplete observations, and parsing anomalies. Ordinary TLS probes cannot collect QUIC.

If canary succeeds while real traffic fails, investigate probes and detection first. Canary capture=not_verified is intentional.

Observations, but no event

Check on: http.request needs request_events, while http.transaction needs transaction completion. Use rules test to inspect where/when, missing fields, cooldown, windows, lateness, and budgets. New window rules warm up and evaluate only after the fixed window closes. Inspect active rule versions, not just disk files.

Plugin load or reload fails

Run a full check and plugin verify. Check ABI, host_schema_version, schema, protocol/port conflicts, hashes, and memory declarations. Non-root analyzers need readable packages and traversable parent directories. Schema/sensitivity changes require coordinated restarts. Old connections retaining an old generation after compatible reload is expected.

Backlog or Dataway failure

Inspect each sink's pending count and failure status. Check the base URL, token_env in the service environment, DNS/TLS, and receiver state. Doctor checks its own environment, not systemd's. 401/403 use long backoff; restart the main service after fixing credentials to retry immediately. Do not include tokens, full request URLs, or raw responses in issue reports.

Restore free space after disk exhaustion while retaining committed WAL data. File rotation retention and queue capacity are separate. If ACK succeeds but retrieval fails, query the destination by event_id and investigate indexing.

TUI shows UNKNOWN / STALE

Check heartbeat freshness, loopback metrics, complete endpoint selection, and status PID versus current service PID. Distinguish insufficient permission from idle traffic. An old snapshot does not establish current health.

Provide useful evidence

Create a diagnostic archive, and include version/support, selected scope, incident time, and sanitized reproduction steps. State whether the issue concerns preflight, synthetic replay, real TLS, or physical capture.