Raw protocol recording

Save captured raw data in a dedicated local folder, selected by protocol, service, or connection. Disabled by default. This archive is independent of the redacted event WAL, NDJSON, and Dataway outputs.

Formats and visibility

Input	File	Contents
network	.pcap	Original packets passing capture filters and associated with recognized TCP/UDP connections, with capture timestamps and link type
endpoint	.stream	Accepted plaintext Protobuf frames, retaining direction, offsets, time, and process attribution; usable by stream replay
Both	Matching .json	OPEN identity, connection, protocol, service, segment sequence, event IDs, completeness, and close state

TLS/QUIC network packets remain encrypted. Endpoint files contain application plaintext streams, not encrypted wire packets or reconstructed PCAP. Both sources can be archived; capture.application_source still selects application observations for rules, without adding duplicate transactions.

Only recognized protocols are retained. Unrecognized, previously dropped, or invalid input is not a complete archive. TCP prefixes before identification are buffered for at most five seconds within the byte budget; isolated final ACKs can fall outside the connection lifecycle. Endpoint recording preserves data actually delivered by the probe and adapter, without recovering earlier capture losses.

Enable a local folder

Merge this fragment into the main configuration. Selected protocols must be installed:

{
  "recording": {
    "enabled": true,
    "directory": "/var/lib/netflowscout/captures",
    "mode": "continuous",
    "sources": ["network", "endpoint"],
    "protocols": ["http", "mysql", "postgresql", "redis"],
    "services": ["billing"],
    "max_file_bytes": 16777216,
    "max_total_bytes": 268435456,
    "max_files": 256,
    "buffer_bytes": 33554432,
    "max_flows": 512
  }
}

Relative directories resolve against the configuration file. The analyzer creates a 0700 directory and 0600 files. Existing directories must belong to the runtime user with no group/other permissions. Use a dedicated folder containing only managed archive files; symlinks and concurrent analyzer writers are rejected. Packaged systemd units allow /var/lib/netflowscout; other paths require analyzer ReadWritePaths and filesystem write permission.

sources defaults to both inputs. protocols is required; ["*"] selects all recognized protocols. Empty services / connection_ids means unrestricted; otherwise values match exactly. dns includes UDP and TCP DNS. Restart the main service and analyzer after recording configuration changes. The TUI recording form and JSON drafts support validation and coordinated activation.

netflowscout check --config /etc/netflowscout/netflowscout.json
sudo systemctl restart netflowscout.service

The full source example is examples/recording.json. It uses the example service and leaves live network capture disabled; configure the real interface or endpoint separately.

Trigger from detection events

For example, retain context only after a particular URL is requested:

rule "export_request" do
  on "http.transaction"
  when http.path == "/admin/export"
  emit "http.export" do
    severity = "warning"
    message = "Export requested"
  end
end

Change the recording mode to:

{"mode":"event","event_names":["http.export"],"pre_seconds":10,"post_seconds":20}

Merge this inside recording, retaining other selectors and budgets. Pre/post values are 0..300 seconds, default zero, measured using analyzer receipt/trigger time. Event triggering requires the live run + analyze pipeline; analyze --pcap rejects event mode. A bounded memory ring holds prehistory. After a matching event commits to the WAL, the analyzer receives a notification, writes available context, and captures subsequent data until the deadline or connection close. Recently closed connections can still be associated. Repeated triggers extend the current connection's deadline; they do not subscribe to future connections. Empty prehistory still enables post-trigger capture for a matching live connection; lost context is marked incomplete.

Per-record events carry connection_id. Connection-grouped windows associate precisely; other windows must group by service to associate currently retained, selected service connections. Windows grouped by neither are skipped and counted in recording_trigger_unscoped_total. Missing prehistory, post windows ending before the connection, and storage failures set incomplete. Event capture does not guarantee a full TCP handshake or application transaction context.

Continuous mode also associates committed events with indexes. Empty event_names associates all event names; a nonempty list filters them. Each segment holds at most 32 distinct event IDs; excess associations mark event_index_limit.

Rotation, quotas, and failures

Rotation occurs per connection at record boundaries, without splitting packets or frames. max_file_bytes defaults to 16 MiB, range 1 MiB..1 GiB. max_total_bytes defaults to 256 MiB, minimum one segment plus 32 KiB, maximum 1 TiB. Each segment reserves 32 KiB for its index. The total budget includes payload and reservations. max_files counts data segments, default 256, maximum 4096.

Only the oldest unprotected closed segments and their indexes are evicted. If active or protected segments occupy the quota, new raw data is dropped and counted without blocking capture. A record exceeding the segment limit is also dropped and marked. The asynchronous input queue and prehistory ring each use at most buffer_bytes, default 32 MiB each; max_flows defaults to 512. Network fragment association has an additional 4 MiB / 1024-group buffer. At most 32 data files stay open, with bounded writer-buffer reuse and one PCAP scratch buffer. Reuse can increase idle resident memory. These are not whole-process RSS limits.

Raw recording is best effort and does not inherit event WAL at-least-once guarantees. Changed data and indexes flush and sync every second; sealing a segment syncs immediately. New associations on closed segments are saved on the next periodic flush or graceful shutdown. Idle segments avoid repeated disk writes. Administrative stops and input disconnects without connection-end notifications mark the archive incomplete. Crash tails can be partial; restart inventories old files without appending, and closed:false indicates an unsealed segment. Trigger notifications use a bounded local socket queue and can be lost on failure or shutdown. Event delivery retains its WAL contract.

netflowscout status --component analyzer --config ... includes a recording object with directory, state, memory/storage accounting, and counters. Analyzer counters include collector_recording_queue_drops, collector_recording_buffer_drops, collector_recording_storage_drops, collector_recording_write_errors, and collector_recording_files_evicted. Main-service counters include recording_trigger_failures_total and recording_trigger_drops_total. File existence alone does not prove complete capture.

retention_hours expires closed segments by age; event_retention_hours automatically protects event-associated evidence against age and quota eviction. Both accept 0..8760 hours, with zero disabling that policy. Expiry runs every minute and at startup. Protection never increases hard quotas: protected evidence can cause new recording drops. Early rotations can expire independently; lookup reports missing segments. Manual pin lasts >0 and <=8760h and covers the whole capture identity, including other episodes and future rotations. Unpin does not cancel automatic event protection. Counters include files_expired and retention_errors. Retention uses local wall-clock time.

To enable recording while generating configuration:

netflowscout configure --service billing --executable /opt/billing/server \
  --recording-dir /var/lib/netflowscout/captures --recording-protocols http,mysql \
  --recording-retention-hours 24 --recording-event-retention-hours 168

Recording is enabled only with --recording-dir. Generated selectors restrict recording to the chosen service and input source. Continuous mode is the default. Event mode additionally requires --recording-mode event --recording-events EVENT_NAME and defaults to five seconds before/after. Analyzer filesystem permissions still apply.

Find and replay

Files use nfs-<capture identity>-000001.pcap or .stream, with .json indexes. Version 2 adds episode, episode_start, and start/end times. Rotations are grouped within an episode; separate event captures on a reused connection never share replay state.

netflowscout recording list --config ./netflowscout.json --event-id EVENT_ID
netflowscout recording export --config ./netflowscout.json --event-id EVENT_ID --out /private/new-evidence
netflowscout recording replay --config ./netflowscout.json --event-id EVENT_ID
netflowscout recording pin --config ./netflowscout.json --event-id EVENT_ID --for 168h
netflowscout recording unpin --config ./netflowscout.json --id CAPTURE_ID

Selectors also include --connection-id, --service, --protocol, and --id CAPTURE_ID:EPISODE. Use --directory for existing archives or exported bundles even when recording is disabled. Export, replay, and protection require an explicit selector. Listing displays index metadata without request/response content.

Active unsealed files cannot be exported: end the connection or stop the analyzer first. Once the writer has stopped, unsealed crash tails remain exportable as unclean_tail/incomplete evidence; malformed tails may fail replay. The destination must be new and outside the managed archive. Directories use 0700 and files use 0600; failure removes the new partial bundle. Original segments and indexes are preserved. manifest.json records UID, time, per-file SHA256, and byte counts for integrity checks, not publisher authentication. Raw bytes bypass redaction.

Replay automatically assembles each episode, retaining one stream OPEN or rewriting a single PCAP header. Missing prefixes, unsealed files, and index/size mismatches are reported. Existing incompleteness propagates into rule records. Repeated OPENs, malformed data, and incompatible link types reject assembly. Each episode uses the formal parser and rules independently. Previews are redacted; scope=per_capture_isolated. Replay never writes the event WAL or delivers events, and does not simulate windows across connections. At most 128 episodes replay at once.

Legacy version 1 streams are split at segment-start OPENs. Legacy PCAP episode boundaries cannot be recovered and segments are conservatively isolated. Evicted original data cannot be recovered.

TUI 9 Recordings offers filtering, completeness details, lookup, replay, export, and protection. 6 Events provides the same actions for a selected event. Export/protection require --control, confirmation, and a writable control audit. 8 Config → recording provides selector, trigger, retention, and quota forms with validation, diff, coordinated activation, and recovery.

Raw content and access

Exact raw bytes can include passwords, Authorization, cookies, tokens, SQL, and personal data. Archives and indexes bypass event redaction; privacy.raw_fields does not control them. Events still pass the existing privacy policy before WAL/output. Enable raw retention only for services requiring it, and handle the folder and backups as sensitive local data. Raw files are never sent to Dataway or included in diagnostic bundles.