Troubleshooting

Start with:

pebble status
pebble diagnose          # [ok]/[warn]/[fail] per check; exit status 1 on any failure

Exit codes#

Status and diagnostic commands (status, diagnose, interfaces, routes, firewall status, dhcp ..., dns ..., vpn status, vpn peers, validate, plan) accept --json.

pebble apply refuses to run#

• "configuration is invalid" — run pebble validate; every error names the configuration path.
• "interface ... does not exist" — check pebble interfaces.
• "/etc/rc.conf line N sets X after Pebble's managed block" — remove that line (or move it above the block); it would override Pebble.
• "would block your current SSH session" — add your client network to management.ssh.allow_from.
• "awaiting confirmation" — pebble confirm or pebble rollback.
• "interrupted transaction ... must be recovered" — pebble rollback.
• "imported ... unconfirmed" — set migration.interfaces_confirmed: true after checking devices.
• "... was rejected by pfctl / unbound-checkconf / dnsmasq" — the native tool's message follows; please report it, as Pebble should never generate invalid files.

Health check failed and the change was rolled back#

The message names the failing check. Common causes:

• Unbound: query ... returned SERVFAIL — check /var/log/messages for unbound errors; unbound-checkconf /usr/local/etc/unbound/pebble/unbound.conf.
• interfaces: igb1 does not have address ... — the device name may be wrong or the driver failed to attach.
• WAN ... has no IPv4 address yet is only a warning: upstream DHCP may be slow or the cable unplugged.

No Internet from the LAN#

pebble firewall test --in lan --src 192.168.10.50 --dst 1.1.1.1 --proto tcp --port 443
pebble routes --lookup 1.1.1.1
sysctl net.inet.ip.forwarding          # must be 1
pfctl -s nat

If forwarding is 0 after a reboot, PF probably failed to load: service pf status, /var/log/messages. Pebble deliberately leaves forwarding off when PF is not running.

DNS#

pebble dns status
pebble dns test www.example.com
drill @127.0.0.1 www.example.com

DNSSEC failures for every name usually mean the clock is wrong: check date and service ntpd status.

"auto-trust-anchor-file ... does not exist" from unbound-checkconf: the trust anchor is missing. pebble apply normally creates it; to do it by hand:

install -d -o unbound -g unbound /usr/local/etc/unbound/anchor
unbound-anchor -a /usr/local/etc/unbound/anchor/root.key
chown unbound:unbound /usr/local/etc/unbound/anchor/root.key
chown root:wheel /usr/local/etc/unbound   # the chroot itself stays root's

IP lists#

pebble firewall iplist status        # last success, entries, last error, whether the ruleset uses it
pebble firewall iplist test 1.1.1.1
pfctl -t pebble_l_doh -T show | head

"never downloaded" means the table is empty: run pebble firewall iplist update. Entries counted as "internal dropped" overlapped your own networks and were deliberately ignored.

WireGuard#

pebble vpn test site-b
wg show

No handshake: check the peer's endpoint and public key, that UDP to the listen port is reaching the WAN (pebble firewall explain), and the clock. Handshake but no traffic: check routes and rules on both sites (see wireguard.md).

A device has no Internet through its VPN exit#

1. pebble vpn status: the exit must have a recent handshake. If it has none, check the endpoint, the provider's public key and that the private key in /usr/local/etc/pebble/keys/wgN.key is the one from the same provider configuration file. Most providers expect persistent_keepalive: 25.
2. pebble firewall test --in lan --src <device> --dst 1.1.1.1 --proto tcp --port 443 should be decided by pebble-vpn-N. If it is decided by something else, the device's reservation does not name the exit, or the device is using a different address: compare its address on the Devices page with the reservation and renew its lease.
3. Names resolve but nothing loads: the provider may only accept DNS from inside the tunnel. Set dns on the exit and renew the device's lease.

A device whose VPN exit is down having no Internet is intended: the kill switch never lets its traffic out through the WAN.

A device is blocked, or not blocked, unexpectedly#

egress: none blocks only Internet destinations; the LAN, the router and WireGuard sites stay reachable. The rule matches the reserved address, so a device keeps its old access until it renews its lease onto that address. pebble firewall explain lists every per-device rule.

The live traffic view is empty#

The Devices page and the per-device view read pfctl -vss. Check that it lists states on the router (pfctl -vss | head). Traffic between two LAN devices never reaches the router and is not shown; a device that is idle shows "idle".

After a crash or power cut#

pebble status shows any unfinished transaction. Boot recovery resolves it automatically before networking starts; otherwise:

pebble recovery boot
pebble rollback

See recovery.md.

Collecting information for a bug report#

pebble version
pebble validate --json
pebble plan --json
pebble diagnose --json
cat /var/db/pebble/transaction.json

These outputs contain no private keys. Review them for addresses you do not wish to share.