Failure bucket configuration

Every failure the Rust core surfaces goes through a closed bucket table (ADR-040): each bucket declares a severity, the surface it taints, and a default action. You can change the action per bucket in ~/.config/degenbot/config.toml:

[failure_policy]
ws_completeness = "degraded"              # keep running through WS drops (operator decision)
"solver_state_desync.missed_log" = "observe"  # reason-level override, quoted dotted key

[failure_policy.sim_failure]              # sub-table form is equivalent to quoting
pre_encode = "quarantine"
revert_economics = "observe"

Actions

Action	Meaning
observe	Metrics only; never surfaces through degenbot.errors.
event	Keyed loud event (deduped per pool, 10-block window); keep running.
quarantine	Loud event + exclude the taint scope from solve resolution; keep running.
exit	Loud event + flush + process exit (fail-fast).

Buckets

Closed set, one row per bucket (from telemetry::error_kind + reason):

Bucket	Default	Overrides allowed?
solver_state_desync.missed_log / .unhandled_reorg / .storage_mutated / .unclassified	quarantine (pool)	yes — containment must be re-declared, never assumed off
solver_state_desync.delivery_lag	event (report-only per ADR-021 Part B)	yes
late_log	event (path) — delivery-jitter lateness, see below	yes
sim_failure.pre_encode	quarantine (path)	yes
sim_failure.revert_pool_state	event (pool, escalate on tripwire corroboration)	yes
sim_failure.revert_economics	observe (benign economics)	yes
sim_failure.rpc	event	yes
ws_completeness	exit	yes — "event" records the drop and keeps running; the resulting state gap surfaces via the desync/quarantine path, and DegenbotProcessDown covers real death
submit_failure / monitor_failure	event	yes
verify_mismatch	quarantine (deny admission)	yes
drain_stall	exit	yes — "event" records + re-arms the watchdog (WARNING: continuing past a stalled drainer means pricing on a frozen clock; intended for bisecting only)

The late_log bucket (HJ5HWF — late-log admission safety)

A forward log that arrives after its block’s D1 tombstone (the first successor log) is delivery-jitter lateness — the expected shape when a tightened settle window (the pump debounce, 50ms → 16ms) publishes a block before its final WS logs cross the quiesce edge, or when the WS feed delivers out of order. The state machine takes the benign late-admit path: the log is dropped un-applied (writers stay confined to the block’s Streaming window — invariant I4), the delivery cutoff the tombstone set never moves (I7), and the pump keeps running.

Lateness is never a fatal, tainted, or tripwire signal:

• the raw rate lands in the dedicated metric family degenbot.late_log.admitted (zero or near-zero on healthy feeds; a sustained rate says the WS reordered, not that the state machine misbehaved);

• the deduped late_log event (one per 10 blocks per endpoint) names the benign path explicitly in its message so a burst of drops under a new settle window can never masquerade as a structural bug;

• it is distinct from degenbot.reorg.recovery_dropped, which counts only silent single-writer duplicates inside an authoritative catch-up’s owned range.

The completeness verify at the tombstone/Published edge (the ws_completeness cross-check) remains the loud safety net for genuinely dropped WS logs — late-but-delivered logs are a disjoint, benign class.

Undeclared buckets (new kinds from an upgrade before you override them) follow the conservative degraded floor and log a warning — they are never fatal and never silent.

Validation and boot behavior

• Config lives in ~/.config/degenbot/config.toml (the same file as otel).

• Keys are closed: kind or kind.reason from the tables above. An unknown bucket is a boot error: the process prints [failure_policy] invalid override - boot refused: unknown failure bucket: "..." and exits 2 before any trading.

• Action values are the four strings above; anything else is also a boot error (unknown failure action: "...").

• Every installed override is logged at boot (level INFO, greppable under failure_policy overrides installed) — a softened quarantine is an operator decision that stays visible.

• Changes require a bot restart; eval happens at import time, before any solve.

How to choose (targets)

• Softer containment only ever buys you uptime, never correctness — quarantined pools are excluded from solve by construction, the solver cannot act on divergent state in any tier.

• exit remains sensible for process-scope buckets; a blind or wedged bot loses money as surely as a dead one, just more quietly.

• Override a tainted bucket only when you have a specific reason (e.g. bisecting a suspected decoder bug with the probe attrs) and monitor the degenbot.quarantine.events + degenbot_engine_quarantined_pools panels on the overview dashboard while it applies.