Events, privacy, and outputs

Event fields

Per-record events include connection_id for association with raw data files. Windows spanning connections do not inherit the first connection identity.

Events include event_id, timestamp_ns, name, rule_id, rule_version, sensor_id, service, protocol, severity, message, and rule-defined fields. Window events can include start/end times. Endpoint events can carry process metadata; a window spanning processes does not inherit the first observation's identity.

Lifecycle events also include alert_key, incident_id, and alert_state, identifying the rule group, incident, and current phase. Retries of the same committed event retain event_id; consumers can deduplicate by that ID.

Redaction by default

SQL, parameters, URL paths, bodies, and other sensitive content can participate in memory-only rule evaluation. Event output defaults to [REDACTED]. Authentication passwords and tokens do not enter output. Raw-field settings cannot recover authentication data discarded during parsing.

If raw auditing is required, explicitly allow each service and input field. For example, merge this into the main configuration:

{"privacy":{"raw_fields":{"billing":["db.statement"]}}}

Allow only necessary fields and restrict file access and retention. Auditing is bounded: truncated means content was clipped; incomplete means capture or correlation lacks context. Neither proves business success.

Local NDJSON

{"file":{"enabled":true,"path":"/var/lib/netflowscout/events.ndjson","max_bytes":16777216,"keep":8}}

Each line is JSON. Confirmation follows a successful write and fsync. Defaults are 16 MiB per file and eight files total; keep includes the current file. Rotation removes delivered older files. Do not run external copytruncate against the active file.

Dataway

{"dataway":{"enabled":true,"url":"https://dataway.example.com","token_env":"NETFLOWSCOUT_DATAWAY_TOKEN"}}

Replace the URL with the actual base address, without a query, user information, or token. Put credentials in root-managed /etc/netflowscout/output.env, mode 0600, injected through the packaged unit's EnvironmentFile. Keep tokens out of JSON, command arguments, and diagnostic archives.

The service posts correctly encoded line protocol to POST /v1/write/logging. Batches are limited to 128 events and 1 MiB of original event data, with a 10-second timeout and no redirects. Failures retry with backoff. 429/503 respect Retry-After; 400/401/403 wait at least five minutes. Restart the main service to retry immediately after fixing authentication.

A 2xx confirms HTTP acceptance, not backend indexing or searchability. Current acceptance uses a local mock receiver; production log retrieval needs separate validation.

Durability and a full queue

At-least-once delivery starts when an event successfully commits to the local WAL. File and Dataway acknowledge independently. Reclamation waits for every enabled sink; lost responses can cause duplicates. Capture, parsing, input, and pre-commit loss are outside this guarantee.

A full queue or disk-write failure rejects new events and increments counters while retaining accepted events. Disabling a sink with unacknowledged events rejects startup. Changing a path or URL redirects pending events to the new destination. Inspect and drain before migrating; deleting the WAL is not a backlog remedy.

For local history filters, process/rule version investigation and per-event acknowledgment receipts, see Setup & investigation.