Flight recorder, health and the terminal dashboard

Status. In use on a FreeBSD 15.1 home router; the log parsers (FreeBSD syslog, RFC 5424 and ISO timestamps) and incident rules are also unit-tested.

What it records#

When monitoring.enabled is true, the agent (pebble_agent) samples the router every sample_interval (default 10 s) and tails /var/log/messages. It records events, not traffic:

It never captures packets and never stores payloads. It records DNS queries only if you turn on dns.query_log (see dns-filtering.md), which keeps a bounded per-device history; turning it off deletes it. Repeated identical log lines within one pass are stored once with a count.

monitoring:
  enabled: true
  sample_interval: 10s
  journal_max_mb: 32      # hard limit on disk
  retention: 30d
  probe_targets: []       # optional IPv4 addresses for the external connectivity check

Storage#

The journal is a directory of JSON-lines segments in /var/db/pebble/journal/. Each event has a stable, increasing ID, the boot ID, the time it was observed (from the log line or sample) and the time it was ingested, a monotonic clock reading, its source, kind, severity, subject (interface, peer or service), message and structured evidence. Writers take a lock; a line torn by a crash is repaired on the next start and counted rather than silently dropped. Segments are deleted oldest first when the size limit or the retention period is exceeded, and pages state from when records are available, so a gap is never presented as "nothing happened".

Traffic history (bytes and packets per interface, never contents) is kept in memory as rings for 5 minutes, 1 hour, 24 hours and 7 days and saved to /var/db/pebble/metrics.json every 5 minutes.

Explaining an outage#

pebble events --last 24h --kind link_down,driver_watchdog
pebble explain outage --last 1h

explain outage (and Events → Explain incidents in the web interface) groups events that are close in time into incidents and applies a small set of evidence rules: watchdog timeout followed by a link reset, link flapping, carrier loss, DHCP offers not received, lease reacquired, address or gateway change, DNS failures, configuration changes, stale WireGuard peers, firewall state changes and crashes. Each finding separates:

• Observed — the events, with their times and sources;
• Likely explanation — what the sequence suggests;
• Not established — what the evidence cannot tell (for example whether a watchdog timeout was caused by the driver, the NIC, the cable or the link partner);
• Suggested next steps;
• separate confidence levels for the sequence and the cause.

Where records are missing (before the oldest event, or across a recorder gap) the report says so instead of reconstructing anything.

Health checks#

pebble diagnose, the overview page and pebble top share one set of checks. Each reports healthy, degraded, unavailable or unknown with the evidence it used and, where useful, what to do. A critical failure (for example the Pebble ruleset not being loaded in PF) is never summarised as healthy. Checks that would need a third party (external connectivity) report unknown unless you configure probe_targets.

Terminal dashboard#

pebble top                 # interactive; 1-4 or Tab to switch views, p to pause, +/- interval, q to quit
pebble top --interval 5s   # slower refresh over a slow link
pebble top --ascii --plain # no Unicode blocks, no bold or dim (also honours NO_COLOR)
pebble top --once          # one plain snapshot (automatic when output is not a terminal)

It reads the router directly through the same collectors as the agent, so it works with the web interface disabled and the agent stopped. Rates are computed from successive kernel counters; after a counter reset it shows no rate rather than a wrong one.