ADR-008: Per-block state machine for the pump’s block clock

Status: implemented (core). Recorded during the block-state-machine review, June 2026. Resolves the verify-race class of bugs (V2-V2-V3 crash at block 25397049: a Mint at 25397047 un-applied when 25397049’s newHeads advanced the cursor while N’s liquidity log was still in-flight). Full design note and rationale live in docs/architecture/block-state-machine.md; this ADR records the settled shape only.

Superseded in part (epic MROOY7, ADR-041): the standalone BlockClock machine was folded into the one StageMachine and the type deleted (hard cutover). The D1 tombstone / D2 quiesce-gate / D3 reorg decisions below survive as machine decisions; read this ADR as provenance, not as a live design.

Implementation status

Implemented in rust/crates/engine/degenbot-bot/src/bot_core/ (commits 5673f8ce, 440b848, c4d21b1e, cdac7363, 0baed1e):

• D1 (tombstone via successor log) — BlockClock + pump wiring. A newHeads header alone never finalizes or drains; only the first removed: false log for N+1 tombstones N. The empty-block finalize fast path (block_pump.rs:593-595) is deleted. Pinned by header_alone_does_not_drain_block + full_lifecycle_tombstone_via_successor_log_then_drained.

• D2 (LogsQuiesced solver-release gate) — the wall-clock DEBOUNCE_MS send timer is replaced by the consume_quiesced predicate. on_send fires only when the open block is quiesced (all dispatched logs applied) AND at a settle point (a DEBOUNCE_MS window with no new event, coalescing a same-block burst into one publish at the tail, OR stream exhaustion). Publication is gated on state, not schedule. A straggler log re-arms the gate (the one-shot consume_quiesced resets; the new log’s receive→apply re-sets it). Pinned by burst_of_logs_publishes_once_at_tail_via_quiesce_gate (pump) + consume_quiesced_publishes_once_per_cycle_and_re_arms_on_straggler (clock).

• D3 (asymmetric late-event handling) — clock returns EnterReorg/ContinueReorg/ CloseReorg/PanicLateForward; the pump routes removed: true to ReorgCoordinator (per-event restore), closes the reorg window on the first removed: false (its block is the new head), and shuts down on a late removed: false. Pinned by reorg_contiguous_chunk_closes_on_first_forward_and_continues + late_forward_log_on_tombstoned_block_shuts_down_pump.

• D4 (backfill single-branch) — backfill_range feeds each fetched log through clock.observe_log (same SM as live logs; no Backfilled → Drained edge). Dead-logs-sub Edge B RPC-budget guard + the dedicated dead-sub detection timeout are deferred (the existing 60s inactivity timeout covers the degraded path).

Deferred to follow-up: the [DIAG] newHeads-stall instrumentation in run_with_stream (operational logging, retained until a follow-up confirms the SM’s liveness paths cover the stall scenario); the dead-logs-sub Edge B timeout tuning + RPC budget.

Context

The pump has one cursor — last_processed_block (= last_drained_block, in solve_coordinator.rs:157) — advanced by two independent WS event sources (newHeads + logs, combined via stream_select at bot_core/block_pump.rs:201). The correctness invariant the verify depends on is:

engine-state@cursor ≡ on-chain@cursor on the liquidity dimension.

Today the cursor is allowed to advance to N+1 (on the newHeads edge) while block N’s liquidity log is still in-flight on the other WS stream. The verify-side “fix” in commit 033a6b2f (drop the redundant startup batch verify) only hides the race from the verify; it does nothing for the solver/dispatch loop, which still runs against possibly-incomplete state. This is a pump-correctness bug, not a verify bug.

The current code enforces the invariant (badly) with scattered booleans — has_logs_this_block, first_header, debounce_active, last_solved_block, current_block — and has_logs_this_block is reset on the next header advance, so a late log silently reopens a “closed” block with no reconciliation. There is no single authoritative “what is the state of block N,” so the invalid transition is easy to fall into and hard to test for.

Decision

Adopt a per-block state machine for the pump’s block clock. One state machine per block number N the pump is tracking; a block is in exactly one state at a time. Only states with observable, non-timer triggers are in the machine.

                 ┌──────────────┐
                 │   Observed   │   ← newHeads(N) OR first log with block_number==N
                 └──────┬───────┘
                        │ first log for N received
                        ▼
                 ┌──────────────┐
                 │ LogsArriving  │   ← ≥1 log received for N; more may come.
                 │              │      Solver MAY run here (fallible, accept re-solve
                 └──────┬───────┘      on straggler, downstream simulation discards)
                        │ first removed:false log for N+1 (the tombstone)
                        ▼
                 ┌──────────────┐
                 │  LogsApplied │   ← N is provably closed by successor opening
                 └──────┬───────┘
                        │ verify completion
                        ▼
                 ┌──────────────┐
                 │   Drained    │   ← last_processed_block() may return N ONLY here
                 └──────────────┘

Transition table (settled)

From	To	Trigger
(none)	Observed(N)	newHeads(N) OR first log with block_number==N
Observed(N)	LogsArriving(N)	first log for N dispatched via dispatch_log
LogsArriving(N)	LogsApplied(N)	first removed: false log for N+1 (the tombstone)
LogsApplied(N)	Drained(N)	verify completion for N

last_processed_block() is permitted to return N only when the per-block machine for N is Drained. Only Drained feeds the cursor.

D1. The tombstone is a real successor log, never a header

LogsApplied(N) is reached via the first log (not newHeads) with block_number == N+1, removed: false. A push WS cannot prove “no more logs for N”; only a successor block’s first real event can. newHeads is demoted to a liveness probe for the logs subscription only, never a completeness signal — a newHeads(N+1) header arriving with no logs for N+1 following within a window indicates the logs subscription may have died (→ D4).

This makes the current code’s two invalid transitions physically impossible (no match arm):

1. Observed/LogsArriving → Drained on newHeads(N+1) — the empty-block finalize fast path (bot_core/block_pump.rs:593-595).

2. Late block_number==N log silently reopening a closed block (has_logs_this_block reset, update_block jumping backward).

D2. LogsQuiesced is a side-channel predicate, not a state

LogsQuiesced was initially drafted as a state between LogsArriving and LogsApplied. It has no honest positive trigger as a state: a push WS has no “no more logs” event, and “queue momentarily empty between two events” tells us nothing useful. So it does not earn a slot in the block lifecycle.

It does earn a slot as a solver-release gate. The debounce timer in the current code is a dumb wall-clock throttle that fires whether or not more logs are queued behind it. Replacing it with a real signal fixes that:

• Per-block in-flight counter: number of logs for N received but not yet fully applied to pool state.

• LogsQuiesced(N) predicate (not a state): true when the in-flight counter for N hits 0 since entering LogsArriving(N). Re-enters (becomes false) on the next straggler log. Semantics: “every log event the WS has given us for N has been fully applied to pool state.”

• Solver-release gate: solver results may be released to consumers only when the current block’s LogsQuiesced predicate is true. The solver may be invoked at any point during LogsArriving; only publication is gated.

The solver may run at LogsArriving (best-effort): a fallible intermediate solve is OK, because these are always simulated later and failing results from an intermediate solve that was superseded later in the block can just be discarded.

D3. Asymmetric late-event handling (their door, not ours)

A log with block_number == N arriving after N is LogsApplied (or Drained) means two very different things depending on the removed flag. A late log on a tombstoned block is either a chain reorg (handle via journal restore) or a broken subscription (halt) — no tolerate-and-re-solve middle path.

• removed: true on a tombstoned block → reorg path.

  • Nodes may deliver reorg events in any order (reverse log index, reverse block number, unordered); we make no ordering assumption. We accept the contiguous chunk of removed: true events.

  • Reorg-window close: gated on the first removed: false event received after entering the reorg path. That event’s block is the new head; forward tracking resumes monotonically from there. No separate resumption-point selection.

  • Pool state restore: restore via the existing ReorgCoordinator::dispatch_reorg_log / BotState::restore_before_block machinery to the common ancestor (the deepest Drained block that survives the reorg). Rewind the per-block machines of unwound blocks to Observed(N-X) and re-track forward. If the reorg is too deep (NoStatePriorToBlock), graceful shutdown — the cursor never regresses past a Drained block.

• removed: false on a tombstoned block → panic/shutdown. A forward log on a tombstoned block means WS delivery is unreliable (out-of-order or duplicated forward events), not a reorg. This is unrecoverable for correctness — we can’t trust any tombstone the WS gave us.

D4. Dead logs subscription → Edge B backfill (taint from N+1, single-branch replay)

The logs subscription may die while newHeads stays alive. This is self-healing in one sense: no forward state transitions occur (no logs arrive → no pool state mutates → nothing corrupts, just goes stale). The recovery is Edge B backfill.

• newHeads(N+1) arrives but no log for N+1 follows within the timeout window → logs sub suspected dead → fire Edge B backfill.

• Backfill range starts at N+1, not N. We only tombstone a block on positive receipt of an event on the next block, so if N is LogsApplied, the logs sub was alive at least through N+1’s first event. The dead-sub gap can only start after the last successfully tombstoned block. N stays sealed.

• Taint range = [N+1, head] — these are the blocks we never proved complete. Mark every block in the range that was LogsApplied/Drained as Tainted, restore pool state to its end-of-N state (the common ancestor), and replay forward.

• Single-branch replay — no fast path. The backfilled logs flow through the same state machine as live logs: N+1 goes Observed → LogsArriving → LogsApplied as logs are fed in. The state-machine overhead is small and one branch means one set of invariants to test. The LogsApplied tombstone arrives via the first removed: false log for the block after the backfill range. (The existing backfill_range at bot_core/block_pump.rs:763 is the same mechanism — there is no distinguished Backfilled → Drained edge.)

Alternatives considered

• Separate cursors — block_clock legitimately leads (the dispatcher’s fee/timing can); only the verify keys off a liquidity_cursor that advances only on reconciled log-application. Smaller blast radius, doesn’t stall the solve clock, but treats the symptom (verify at a non-moving point) rather than the disease. Rejected: the solve path would still run against possibly-incomplete state — exactly the lag that burned the verify thread. The SM is only the right call if the solver itself is provably running against complete liquidity state, which is what this bot requires.

• LogsQuiesced as a state with a debounce-timer trigger — promotes LogsArriving → LogsQuiesced on “WS silent for ≥ debounce window.” Rejected: ties state transitions to a wall-clock timer, which makes no sense during a fast multi-block reorg/replay (would cascade O(debounce × range) of wall time before catching up). The debounce timer is a solver-to-consumer release throttle, a separate concern from the state machine — see D2.

• Distinguished Backfilled(N) → Drained(N) edge that bypasses LogsApplied only when the log set came from a self-contained eth_getLogs fetch. Rejected: the distinction “WS contribution is racy, RPC contribution is authoritative” would make correctness depend on the source of the log rather than the machine’s invariants. Both flow through the same lifecycle, because the invariants (tombstone via successor, verify-sealed Drained) are what establish correctness, not the log source.

What this does NOT supersede

The per-pool pin gates (step-1 / step-2 snapshots, V3 + V4 twins) complement the SM, they do not replace it:

• The pins are the verify’s defense (frozen-block comparison, immune to whatever the cursor is doing).

• The SM is the pump’s defense (the cursor itself can’t advance past un-applied logs; the solver can’t publish against un-applied state).

They defend different things and should both exist: the pins catch a bug the SM should have prevented but didn’t (defense in depth); the SM prevents the solver from running against incomplete state in the first place (the thing the pins can’t help with, because they only run at verify time, not solve time). V4 verification already pins step-1/step-2 via the post_drain_snapshot (twin of V3’s), so the cursor-correctness fix transfers without family-specific work — the SM is family-agnostic (it’s about block completeness, not pool math).

Deferred / open

• Multi-engine fan-out interaction. SolveCoordinator fans out on_drain/on_send to every engine under the drain_lock. The per-block SM must live above the coordinator (one machine per block, shared across engines), not per-engine — otherwise a slow engine could hold the cursor behind a fast one. Design the SM as a sink wrapper, not an engine field.

• Edge B RPC budget. Edge B is one eth_getLogs([N+1, head]) per dead-sub timeout. On a healthy node this is ~0 fires; on a laggy/dropping subscription it’s the difference between a correct bot and a silently-trading-against-stale-state bot. Worth it, but needs a budget so a misbehaving node can’t DOS the bot with perpetual reconciliation. Specific budget value deferred to implementation.

• Timeout value for dead-logs-sub detection. The window after which newHeads(N+1) with no logs for N+1 implies the logs sub died — deferred to implementation (tuned ~2× observed WS log lag, as in the design note).

References

• docs/architecture/block-state-machine.md — full design note and rationale.

• docs/adr/ADR-006-bot-as-per-chain-orchestrator.md — Bot as state owner; the SM is the per-block clock layer above SolveCoordinator.

• rust/crates/engine/degenbot-bot/src/bot_core/block_pump.rs (line refs in the design note) — current scattered-boolean invariants the SM replaces.