ADR-041: The block-epoch pipeline — one stage machine over a cheap-read data plane

Status: implemented (2026-09-07, ergo epic MROOY7 / task PLRGIN). Settled in the originating architecture conversation, recorded here so every later task can execute without it; the user checkpoint on this ADR was sign-off of the stage table and seam retirement list (canonical form in the design doc). Implemented across the epic’s tasks and validated by the final-integration gate: capture-replay regression sweep (zero divergences; see PLRGIN result) and the live Jaeger soak A/B against the pre-epic operator baselines (../architecture/stateview-feasibility.md §3). The unified StageMachine lives at rust/crates/engine/degenbot-bot/src/bot_core/stage_machine.rs (ergo 7NFYQW); the retired DrainSink/Engine/SolveCoordinator/ DispatchOwner/DirtySets/EngineSubscriber seams are gone (SZJUKL).

Context

The per-block loop of the arb engine is currently choreographed by six correlated state machines plus a parallel accounting system built around them:

• EnginePhase, BlockClock (ADR-008), PumpFSM (ADR-028), the registration verify-lifecycle (ADR-022), the path lifecycle, and the delivery lifecycle — living in rust/crates/engine/degenbot-bot/src/bot_core/{block_pump.rs,pump_fsm.rs,block_clock.rs} and rust/crates/engine/degenbot-bot/src/arb_engine/{mod.rs,lifecycle.rs,path_lifecycle.rs,delivery_lifecycle.rs}. Each is pure (the ADR-028 producer/driver family), but their interaction is opaque: which machine may advance in a given block, and in what order, has no single owner.

• A parallel accounting layer — the DrainSink/Engine dual seam, SolveCoordinator with its drain_lock, the DispatchOwner + DrainWork FIFO, DrainerHealth, the DirtySets + EngineSubscriber classification, and a soup of anchors (solve/sim/verifier/backfill anchors, last_solved_block, last_drained_block, BlockMetadata) that each carry an answer to “what block is this work about?”.

• The engine Mutex sits on the solve path (sharded by ADR-037 for contention, but still an ordering constraint: drain_lock → engine Mutex → BotState RwLock).

The state-accuracy tripwire (ADR-021) exists largely because nothing structurally prevents a desynced snapshot from reaching the solver. A design in which the data plane’s writers are stage-confined makes the in-process half of that concern unrepresentable rather than merely detected.

An ergo spike (KWKEVV) already settled the data-plane mechanism question (see the stateview feasibility doc recorded by that spike; canonical copy on branch pi-fabric/stateview-spike). This ADR records the pipeline architecture that spike unblocked.

Decision

1. One pipeline stage machine per block epoch

A single stage machine owns every per-block edge condition. The canonical stage sequence is:

Streaming → Quiesced → Resolved → Solved → Simulated → Gated → Published → Finalized, plus Rewind{to_epoch} on reorg.

Rewind is a transition from any stage to a fresh epoch at an earlier block; the epoch’s seq bumps and stale contexts fail fast. Full stage responsibilities, data-plane posture, and absorbed sub-state are specified in the design doc.

The six existing machines fold in as sub-state of the stage machine; their pinned tests are the behavioral contract and are ported verbatim. Runtime decisions are dispatched through a single StageHandlers seam — the pure per-stage function set the runtime drives — completing the ADR-028 pure-producer/thin-driver pattern at pipeline scope: BlockPump becomes an event+tick feeder and decision executor only. Its reorg-episode and resume tracking (block_pump.rs) folds into Rewind handling, and its WS transport

• watchdog machinery later moves to degenbot-ingestion (epic task 5WTYYQ).

2. The data plane survives as cheap-read (spike KWKEVV outcome)

Cheap-read (mechanism (a)) for every family: V2-family scalars, V3 tickmaps (Uniswap/Pancake/Sushi), and V4 tickmaps. No materialization, no COW. Concretely:

• StateLock<RwLock<BotState>> stays the data plane. Writers are confined to the Streaming stage. Quiesced, Resolved, Solved, and Simulated hold read guards; with the engine Mutex off the solve path, those reads are uncontended by construction.

• Measured basis (full method and repro in the feasibility doc): 90th-percentile CL tickmap density is 2–4 ticks (registry max 1,536); a full tickmap clone is 40–50 ns at p90 density and 16–19 µs p99 at the registry max; journal-replay of a full 32-block rewind is ≤ 23 µs p99 even at 8 priors/block; Arc COW view construction is 30 ns — all ~10⁵ under the 2 ms view-construction leg. The cheap-read exception leg did not trigger: per-path solve p99 is 5 ms and the solve gate alone 1 ms (≪ the 100 ms threshold), and live state_lock_wait p99 is ≤ 0.1 ms across 19.6 M acquisitions — measured on the still-contended, pre-stage-machine architecture. Since cheap-read only loses if both rule legs pass, cheap-read wins outright for all families.

• Sizing for Rewind: the restore_before_block bounds (≤ 23 µs p99 V3 at depth 32; 20–30 ns V2) mean no view machinery is needed to hit the reorg budgets. StateLock hold/wait histograms are retained as the budget verifier. One out-of-corpus caveat is recorded: whole-map clone goes superlinear above ~131k entries (≥ 16 MB working sets, allocator/TLB) — irrelevant at today’s registry max of 1,536 ticks/pool.

3. Seam retirement

The parallel accounting system is deleted, not wrapped (hard cutover, Q6). Exact list, with disposition:

1. DrainSink/Engine dual seam → one StageHandlers seam. The pump’s executor fan-out (ADR-028 addendum (c)) becomes one trait; delivery and submission subscribe at the Published edge as sinks, not as a dispatch seam in front of the engine.

2. DirtySets + EngineSubscriber classification → EpochDelta. Log application records touched pools as a byproduct (the delta rides the BlockContext); affected-path derivation reads the delta directly behind a capture-replay parity gate.

3. SolveCoordinator, drain_lock, DispatchOwner, DrainerHealth dissolved. They accounted for work the stage machine now owns; the no-progress watchdog obligation moves onto the machine’s watchdogs.

4. Engine Mutex off the solve path. ADR-037’s sharding made it low-contention; stage confinement removes it as an ordering constraint entirely. Registration/FFI locking via StateLock remains (a slow operator path, not the solve path).

5. Anchor soup → a single Epoch carried on BlockContext. Solve/sim/ verifier/backfill anchors, last_solved_block, last_drained_block, and BlockMetadata are replaced by exactly one block coordinate; “what block is this work about” has one answer.

4. ADR-021 repositioning

ADR-021’s posture (detect, classify, stop loudly, never heal) is retained and referenced, but its mechanism repositions: in-process desync detection (solver-state vs chain scalar comparison) retires — desync becomes unrepresentable once writers are stage-confined — while RPC-disagreement verification at the Published edge is retained (the chain/RPC truth can still diverge from us; that failure stays loud and means the provider diverged, not our bookkeeping).

5. The Q1–Q6 decision record

Q	Decision
Q1	A+D hybrid confirmed — one stage machine (A) driving a stage-confined cheap-read data plane (D)
Q2	Incremental in-place throughout. A temporary internal A/B gate is permitted only inside the stage-machine swap task, and must be removed by that task’s completion
Q3	StateView mechanism was gated on the spike outcome; spike KWKEVV closed the gate: cheap-read for all families
Q4	Arb engine ships first. Multi-engine machinery is a non-goal; the design is guarded by a NoopStubEngine exercising every StageHandlers hook (see non-goals)
Q5	12-factor config parity (file + env, one typed BotConfig); effort for coherence is sanctioned
Q6	Immediate cutover on each task’s completion — 0.6 is alpha, breaking changes permitted. ADR-010/011 Alembic gating and the 0.7 kill list are untouched by this epic

6. Runtime lifecycle is an orthogonal axis

Pipeline stages answer “what has happened to this epoch’s block”. The process-level runtime lifecycle — Boot → Subscribed → SnapshotLoaded → Resumed — is a separate axis and never interleaves with the stage machine’s per-epoch transitions. A resume starts a fresh stage machine at the snapshot-seed epoch; the sweep-back of epoch bookkeeping is what SnapshotLoaded exists to bound.

EnginePhase is remapped in full onto this axis: its role — registration/snapshot/backfill ordering — is runtime-level, not per-epoch, so it contributes no Solved/Simulated sub-state to the stage machine. The existing enum (Created → Subscribed → SnapshotLoaded → (Backfilled) → Resumed) already carries exactly this shape.

Consequences

• One decision surface at pipeline scope. Exactly one component emits drain, publish, finalize, notify, backfill, recover, and verify decisions; the six machines are its sub-state. The ADR-008/ADR-028 pinned tests are the contract and are ported verbatim.

• Composition desync goes away structurally, so its detector is deleted. The ADR-021 in-process tripwire surface retires; upstream verification at the Published edge remains.

• Cheap-read means near-zero new machinery. No view representations, no COW memory, no replay cost on the steady path; the ReorgJournal already in the pools crate bounds Rewind.

• NoopStubEngine is the executable spec of hook completeness. It asserts stage order, epoch monotonicity, and rewind handling so a future second engine cannot discover a missing hook at integration time, and keeps the StageHandlers trait honest from day one.

• Tracing carries epoch on every span/metric, so Jaeger/Grafana tell the migration’s regression story directly.

• Migration order mirrors the epic task graph (epoch types → EpochDelta → StateView plane → StageHandlers + stub → unified machine → telemetry → seam retirement → ingestion crate → final integration + status flip).

Non-goals

• No multi-engine machinery. One arb engine exists; no registry, no routing, no second-engine configuration surface. The NoopStubEngine is a test-declared conformance harness — a test-landmine guard, never runtime-selectable.

• No new StateView representation. The spike’s cheap-read outcome retires the COW/journal-materialization mechanisms; their costs remain recorded as sizing input for Rewind and out-of-corpus growth only.

• No auto-repair. ADR-021’s posture is unchanged: divergence and the desync class stop loudly; nothing self-heals state from chain reads.

• No backwards-compatibility layer. Per the repository architecture and Q6, retired seams are deleted with a hard cutover; no legacy exports are maintained in parallel.

• No drift into the 0.7 ADR-010/011 kill list. Alembic retention, the SQLAlchemy models, and ensure_schema’s Alembic branch are out of scope for this epic entirely.