Pebble / Docs / Operating

Transactions, rollback and recovery

What `pebble apply` does#

Takes an exclusive lock (/var/db/pebble/lock, flock; released automatically if the process dies).
Refuses to start while another deployment is pending or unresolved.
Validates the configuration and checks the live machine (interfaces, key files, time zone, rc.conf overrides, your current SSH session).
Generates all files and computes the changes. No changes: stop.
preparing — stores the candidate as a numbered generation (/var/db/pebble/generations/NNNN) and a snapshot of everything it may touch: every owned file (or the fact that it did not exist), the whole of /etc/rc.conf, host name, time zone and owned routes, plus informational captures of ifconfig -a and the routing table.
Builds the DNS blocklist from cached feeds, then validates the candidate files natively: pfctl -n -f, unbound-checkconf, dnsmasq --test, sh -n on the spliced rc.conf. Any failure aborts with nothing changed.
For disruptive changes (network, firewall, WireGuard, web interface), arms rollback before touching anything: a cron entry that checks every minute and a detached timer started with daemon(8). Either alone is sufficient.
applying — writes files atomically and runs operations in dependency order: host name and time zone, PF ruleset, IPv4 forwarding, interfaces, default route, WireGuard, static routes, dnsmasq, Unbound, sshd, ntpd.
Health checks that test function, not just exit codes: PF enabled, forwarding on, interface addresses present, WireGuard interfaces exist, routes present in the kernel, dnsmasq running, and Unbound actually answering a query for the router's own name. A failure rolls back immediately.
pending (disruptive) — waits for pebble confirm until the deadline; or committed (non-disruptive).

pebble confirm commits, removes the cron watchdog and prunes old generations (generation 1, whose snapshot is the pre-Pebble baseline, and the current generation are always kept).

State machine#

            ┌──────────── abort (validation failed / crash) ─────────┐
            │                                                        ▼
 (new) ─► preparing ──► applying ──► pending ──► committed        aborted
                           │  │         │
                           │  └─(non-disruptive)──► committed
                           ▼            ▼
                       rolling_back ◄───┘ (deadline, reboot, failure, 'pebble rollback')
                           │   ▲
                           │   └── (crash: retried; restoration is idempotent)
                           ├──► rolled_back
                           └──► failed (needs an administrator; 'pebble status' says so)

The journal (/var/db/pebble/transaction.json) is rewritten atomically (temporary file, fsync, rename, directory fsync) at each transition.

Found in state	Recovery
preparing	discard the candidate: nothing live changed
applying	roll back to the snapshot
pending	roll back (deadline passed, reboot, or explicit rollback)
rolling_back	roll back again
committed, rolled_back, aborted	nothing
failed	nothing automatic; administrator intervention

What protects you#

Failure	Mechanism
SSH session drops	`apply` ignores SIGHUP; the timer and cron are independent processes
`pebble` crashes or is killed	cron sees `pending` past the deadline, or `applying` 5 minutes past it, and rolls back
Firewall locks you out	not confirmed → rolled back by timer or cron
Wrong interface or route	same; plus the live check refuses a policy that blocks your current SSH session
DNS or DHCP fail to start	health check fails → immediate rollback
Reboot while pending	`rc.d/pebble_recovery` runs before `netif`/`pf`, restores the snapshot, and makes `rc(8)` re-read `rc.conf` (SIGALRM), so the machine boots into the previous configuration
Power loss while writing	every file is replaced atomically; the journal says how far the transaction got
Ruleset fails to load at boot	`rc.d/pf` loads the generated fallback (DHCP + SSH from the LAN only); forwarding stays off

What is not atomic#

Be aware of the limits:

The set of files is not replaced in one atomic step; each file is. The journal and snapshot make the set recoverable.
Live operations (restarting an interface, reloading PF) take effect immediately; rollback re-runs the inverse operations. Connections may be interrupted during either.
DHCP leases handed out and DNS cache contents during a failed deployment are not rolled back.
Restoring the snapshot restores /etc/rc.conf byte for byte: an unrelated edit made while a deployment was pending is lost on rollback.
The DNS blocklist is data, not part of generations; it is kept with its own last-known-good copy.

Commands#

pebble apply [--rollback-after 5m] [--confirm-always] [--dry-run]
pebble confirm
pebble rollback                   # undo the pending deployment, or redeploy the previous generation
pebble rollback --generation 3    # redeploy a stored generation (asks for confirmation as usual)
pebble status                     # current generation, pending deadline, failures

Generations contain only generated files and manifests (no keys). pebble backup additionally archives pebble.yaml and pf.local.conf (and keys only with --include-keys: those archives are not encrypted, so store them on encrypted media and rotate the keys if a copy leaks).

Manual recovery#

If everything else fails, from the console:

cat /var/db/pebble/transaction.json
ls /var/db/pebble/generations/          # NNNN/before/ holds the snapshot
pebble recovery boot                    # resolves any unfinished transaction at file level
service pf onerestart                   # or reboot

To return to the state before Pebble was first applied, the files in /var/db/pebble/generations/0001/before/ (listed in its manifest.json) are the originals, including the original /etc/rc.conf.