The block-epoch pipeline

Ergo epic MROOY7 · decision record: ADR-041 · data-plane spike outcome: the stateview feasibility doc (spike KWKEVV; canonical copy on branch pi-fabric/stateview-spike)

Implemented (2026-09-07, ergo epic MROOY7 / PLRGIN). The tables below are the signed-off design the epic executed against; As-built at the end of this file maps the landed architecture onto the real types and files.

This is the working design reference for the block-epoch pipeline: how one block flows from logs to settled results through a single stage machine, over a single seam between the pump and the engine. Every implementation task of the epic builds against the stage table and seam retirement list below, which the user checkpoint has signed off; code lands against them as the epic’s tasks execute.

There are two orthogonal axes:

1. Runtime lifecycle — process-level phases: Boot → Subscribed → SnapshotLoaded → Resumed. It never interleaves with per-epoch work; see its table below.

2. The block-epoch stage machine — the per-epoch stages of one block, below.

The existing PumpPhase enum (formerly EnginePhase; Created/Subscribed/SnapshotLoaded/Backfilled/Resumed) is remapped in full onto axis 1: its role — registration/snapshot/backfill ordering — is runtime-level, not per-epoch, so it contributes no sub-state to the Solved/Simulated stage rows.

Epoch invariants

• I1. Epoch { block, seq } is the only block coordinate. BlockContext carries it; every stage transition, decision, span, and metric records it. The anchor soup (solve/sim/verifier/backfill anchors, last_solved_block, last_drained_block, BlockMetadata) is deleted.

• I2. seq is monotone non-decreasing; it bumps exactly once per Rewind. block may regress on reorg; seq never does. (block, seq) pairs compare by seq first, then block.

• I3. A BlockContext whose (block, seq) does not match the active epoch fails fast — mirroring the repository’s fail-loud posture (ADR-021): stale contexts are never silently applied.

• I4. Writers to StateLock<RwLock<BotState>> exist only in the Streaming stage of the active epoch. Quiesced..Simulated take read guards (cheap-read, per the spike). Gated..Finalized commit only delivery/results surfaces, never pool state.

• I5. Exactly one Publish per quiesce cycle; Published implies the block’s completeness verdict (tombstone-by-successor or settle) was obtained.

• I6. Rewind{to_epoch} may originate from any stage; at most one rewind is in flight, and a second fails fast.

• I7. The delivery cutoff (glossary: Last complete block, on BotState) is monotone and never reset by a resume (ADR-028 addendum (a)); it is mirrored from the Finalized stage’s tombstone verdict.

Stage table

Stage	What happens	Data-plane posture (StateLock<RwLock<BotState>>)	Emits / decides	Sub-state absorbed (from the six machines)	Retires
Streaming	Live WS log events + gap eth_getLogs backfill applied in log order	The only stage permitted to write	Dirty pools into the epoch’s EpochDelta; recovery single-writer discard; Backfill request	BlockClock log arrival; PumpFSM recovery rules (ADR-028)	DispatchOwner’s dispatch-log path
Quiesced	All dispatched logs for the open block applied; completeness classified (tombstone vs settle; early-slice/debounce gates)	Read guard	Notify (header tick), Verify (WS completeness), Backfill, watchdog Recover/LogSilence — data, not timers	BlockClock tick detection + quiesce classification (ADR-008); PumpFSM watchdogs + backfill trigger (ADR-028)	inline quiesce/block-window reasoning
Resolved	The block’s EpochDelta → affected-path derivation	Read guard (path index)	The epoch’s affected-path set	PathLifecycle registration bookkeeping	DirtySets + EngineSubscriber classification
Solved	Solver runs the affected paths	Cheap-read: read guard through the solve — solve_path_duration p99 5 ms, solve_gate_duration p99 1 ms (budget 100 ms)	Solved candidates	Solver dispatch	drain_lock + engine Mutex on the solve path (ADR-037 shard retired here)
Simulated	In-process revm simulation — the sole executor (ADR-019)	Read guard	Derivation/profit facts per candidate (tri-state per ADR-030)	simulation bookkeeping only (EnginePhase carries none of this — it is runtime-lifecycle, see below)	the sim anchor of sim_anchor.rs (expressed as Epoch)
Gated	Risk/size/profit rechecks on solved+simulated candidates	None (pure evaluation)	Deliver/reject verdicts per bucket (ADR-040)	DeliveryLifecycle gating policy	DrainerHealth no-progress checks (→ machine watchdogs)
Published	Winning bundle to execution; sinks subscribe here (delivery channel, submission, Python)	None (results written to delivery/FFI surface, not pool state)	Publish — exactly one per quiesce cycle; RPC-disagreement Verify (ADR-021 upstream check retained)	PumpFSM settle publish gate	DrainWork::Publish + DispatchOwner + DrainSink routing
Finalized	Epoch closed: delivery cutoff stamped, results final	Cutoff monotone on BotState; never reset by resume	Epoch close	DeliveryLifecycle terminal stamps; registration verify-lifecycle terminal (ADR-022)	cursor stamps in solve_coordinator.rs (last_solved_block)
Rewind{to_epoch}	Reorg unwind from any stage to a fresh epoch at an earlier block	ReorgJournal restore-before-block ≤ 23 µs p99 (V3, depth 32), 20–30 ns (V2); no view machinery needed	Epoch.seq bump; stale contexts fail fast; epoch views above the fork invalidated	The pump’s reorg-episode and resume tracking (block_pump.rs)	the ReorgCoordinator becomes the rewind executor

The pump’s (block_pump.rs) reorg-episode and resume tracking folds into Rewind handling; its WS transport + watchdog machinery later moves to degenbot-ingestion (epic task 5WTYYQ).

Runtime lifecycle (orthogonal axis)

Phase	Meaning	Stage-machine relation
Boot	Process init: config, telemetry, DB	No stage machine exists
Subscribed	WS subscriptions + topic/address filters up (degenbot-ingestion)	Streams feeding; no active epoch
SnapshotLoaded	Snapshot seed epoch E(S) established from the store	Fresh stage machine; gap [S+1, W] backfilled and owned by backfill
Resumed	Live log flow enters Streaming	The per-epoch stage cycle runs; delivery-cutoff monotony preserved across resumes

EnginePhase is the in-code representation of this axis, remapped here from any per-epoch reading; its registration/snapshot/backfill ordering role is runtime-level, never per-epoch stage sub-state.

Epoch invariants at the stage boundaries

See invariants I1–I7 above; the spike-derived numbers binding the stage table are: writers confined to Streaming (log-burst window p99 ≤ 100 ms); state_lock_hold/state_lock_wait p99 ≤ 0.1 ms across 19.6 M acquisitions on the still-contended pre-stage-machine architecture; rewind bounds above.

Seam retirement list

The parallel accounting system is deleted, not wrapped (Q6: hard cutover):

1. DrainSink/Engine dual seam → one StageHandlers seam. One trait; a future second engine implements it or there is no second engine. NoopStubEngine is the executable spec keeping the trait honest (see non-goals).

2. DirtySets + EngineSubscriber classification → EpochDelta. Log application records touched pools as a byproduct of dispatch; affected-path derivation reads the delta. EngineSubscriber shrinks to liveness + notification only.

3. SolveCoordinator / drain_lock / DispatchOwner / DrainerHealth dissolved. The stage machine owns the edge conditions; delivery and submission are sinks at the Published edge, not seams in front of the engine.

4. Engine Mutex off the solve path. Stage separation makes the cheap-read on Quiesced..Simulated uncontended by construction. Registration/FFI locking via StateLock remains (slow operator path, not the solve path).

5. Anchor soup → a single Epoch on BlockContext. solve/sim/verifier/ backfill anchors, last_solved_block, last_drained_block, and BlockMetadata collapse into one Epoch on BlockContext.

Non-goals

• Multi-engine machinery. No registry, no routing, no second-engine configuration surface. Landmine guard: NoopStubEngine implements StageHandlers alongside the real arb engine and is exercised in a scripted conformance harness (synthetic block stream driving the full lifecycle + a reorg + a backfill episode, asserting hook completeness, stage order, and epoch monotonicity — Q4). It is test-declared: a conformance harness, never runtime-selectable.

• Materialization / COW StateView machinery. Cheap-read won unambiguously (spike KWKEVV); the alternative mechanisms’ costs are retained only as Rewind-bounding data.

• Tripwire re-expansion. In-process desync detection retires (desync unrepresentable post stage-confinement); the upstream, Published-edge RPC-disagreement check stays and stays loud (ADR-021 posture).

• Backwards compatibility of any retired surface (hard cutover, Q6). Pinned behavioral tests are ported; stale APIs do not get a parallel life.

• Schema changes. ADR-010/011 Alembic ownership and the 0.7 kill list (see the repository AGENTS.md) are untouched by this epic.

Migration order (mirror of the epic task graph)

#	Ergo task	Lands	Stage coverage
1	7LKJFY ✅ + KAHU5W	Typed BotConfig + 12-factor parity; migrate env reads	(config axis; orthogonal)
2	KWKEVV ✅	StateView spike — cheap-read for all families; Q3 gate closed	data plane
3	T6IYKY	Epoch + BlockContext; delete the anchor soup (seam #5)	context for all stages
4	LXDY4C	EpochDelta dirty tracking; delete DirtySets + subscriber classification (seam #2) under a capture-replay parity gate	Streaming → Resolved
5	2UVG3E	StateView data plane: write confinement + engine lock off the solve path (seam #4); reposition the ADR-021 tripwire	Streaming ↔ Quiesced..Simulated; Published (Verify) — landed (cheap-read branch): DEGENBOT_DETACHED_SOLVES default ON (engine Mutex off the solve path); the in-process chain-vs-solver tripwire module deleted; EpochDelta placeholder unified onto the real ledger; lock inventory + p99 replay in the feasibility doc §5.1
6	YM2FZR	The StageHandlers trait + NoopStubEngine conformance harness (target shape of seam #1)	all (trait shape)
7	7NFYQW	Unified stage machine: fold the six machines, BlockPump → thin driver, pinned tests ported verbatim; Q2’s only sanctioned internal A/B gate lives here and must be deleted by task end	all
8	BF43PM	Per-stage OTel spans + metrics carrying epoch attributes	observable across all
9	SZJUKL	Seam retirement: delete DrainSink/Engine/SolveCoordinator/drain_lock/DispatchOwner/DrainerHealth (seams #1, #3)	Published-edge + drive wiring
10	5WTYYQ	Extract degenbot-ingestion (pyo3-free); Python becomes a Published-edge sink	Subscribed + Published
11	PLRGIN	Regression: capture-replay sweep + live Jaeger soak A/B; flip ADR-041 to implemented; update CONTEXT.md vocabulary	proves all
The order above is dependency-ordered per ergo; #8 (BF43PM) and #9
(SZJUKL) may interleave once #7 lands.

As-built (post-MROOY7)

Every type and file below is verified to exist in the merged tree (epic MROOY7 landed through PLRGIN; see ADR-041, status implemented). Task ids referenced: typed config KAHU5W, data-plane spike KWKEVV, epoch contexts T6IYKY, ledger LXDY4C, lock/confinement 2UVG3E, trait YM2FZR, machine fold 7NFYQW, stage telemetry BF43PM, seam retirement SZJUKL, transport extraction 5WTYYQ, final integration PLRGIN.

Crate map

degenbot-ingestion — the pyo3-free WS transport (boundary contract: ingestion emits, the runtime decides — the crate knows nothing about BotState or the stage machine). Source: rust/crates/integrations/degenbot-ingestion/src/

• ingestor.rs — WsIngestor: one WS connection; newHeads + unfiltered logs merged into one IngestEvent stream (stream_select, fair interleave). subscribe_with_handshake → SubscribeBoundary — the MJXP5Z one-stream handshake (no drop+resubscribe; handshake-consumed logs are re-injected), with the DFQYM5 log-liveness boundary (LOG_CATCHUP_SETTLE_SECS = 15 settle window before the header-confirmed fallback). Gap backfill = fetch_logs over eth_getLogs in DEFAULT_BACKFILL_CHUNK_SIZE (2000-block) chunks; BACKFILL_TIMEOUT_SECS (60) is both the idle/degraded window and the handshake deadline. exact_relevant_indices is the client-side exact-topic pre-filter the WS-completeness cross-check uses (the server-side OR-list over-matches on some nodes).

• events.rs — IngestEvent (BlockHeader | Pool); PoolEvent carries { epoch, log_index, payload } so consumers order/drop without touching the payload.

• topics.rs — RELEVANT_TOPICS, is_relevant_log: the single Rust-side hot pre-filter (also the backfill filter’s OR-list source and the dispatcher’s defensive re-check).

• filter.rs — build_backfill_filter / backfill_filter (single-block shape).

• watchdog.rs — Watchdog: the transport liveness windows (HEADER_STALENESS_SECS = 30, LOG_SILENCE_SECS = 60, LOG_WAIT_MAX_AGE_SECS = 5 — the SONJQA span-force-close bound) + per-episode silence_alarm_count. The decisions on those windows belong to the stage machine (watchdog_phase).

• examples/headless_boot.rs — the standalone no-Python boot smoke (just test-standalone runs it).

degenbot-bot — the runtime (the stage machine, its driver, and the engine):

• bot_core/stage_machine.rs — StageMachine (7NFYQW): the ONE pure, I/O-free machine over one block epoch — no provider, no timers, no Instant, no locks. The six retired machines (incl. the standalone BlockClock, PumpFSM) are folded in as sub-state (hard cutover). Emits HeaderDecision, LogDecision, StageDecision, CompletenessDecision; exposes watchdog_phase (Healthy/HeaderStale/LogsSilent) — the phase space the dissolved no-progress accounting mapped onto. Invariants I1–I7 (above) are pinned by the ported tests.

• bot_core/epoch.rs — Epoch { block, seq } + BlockContext (T6IYKY): the only block coordinate. The derived Ord is (seq, block) — a rewind sorts ABOVE any earlier-generation epoch (a monotone cursor never regresses across a rewind even though the block moves down); ensure_current fail-fasts stale contexts with StaleEpoch.

• bot_core/epoch_delta.rs — EpochDelta (LXDY4C): the per-epoch touched-pool ledger keyed by AffectedKey (degenbot-solvers), recorded as a byproduct of Bot::dispatch_log; take_keys is the drain’s atomic consumption; a rewind RELABELS the ledger (set_epoch) and retains keys (solve-cursor state, not block-window state).

• bot_core/stage_handlers.rs — StageHandlers (YM2FZR): the one engine seam, encoding the stage table as a required-hook trait (no default bodies: adding a hook without updating every implementer is a compile error) plus the exhaustive Stage order table (legal_successors, sized ALL_STAGES).

• bot_core/block_pump.rs — BlockPump: the thin async driver. Work executes INLINE at the machine’s decision points (no FIFO); it feeds Bot::dispatch_log per log, drives the stage hooks, and runs the driver-side reorg_flying_stale I3 check at each work site.

• bot_core/log_dispatcher.rs — LogDispatcher: decoder registry + dispatch (decode → apply under a write guard → release → notify PoolStateSubscribers; records into the epoch’s EpochDelta as a byproduct). Strict DEGENBOT_WS_COMPLETENESS mode fails loudly on malformed-event drops. Span: degenbot.log.dispatch.

• bot_core/reorg_coordinator.rs — ReorgCoordinator: per-event journal rollback (idempotent + order-insensitive restore-before-block); NoStatePriorToBlock → the pump shuts down gracefully (never a silent stale state). Rewind is a Bot concern, never a stage-hook seam.

• bot_core/solve_anchor.rs — SolveAnchor: the request block floored by the pool-state head, itself an Epoch (the head-floor desync rule from MQIZ5M/IIA/ 0x99ac8c).

• bot_core/state_lock.rs — StateLock: the diagnostic parking_lot::RwLock wrapper (Z4Z6VO) guarding BotState; hold-tracking forensics off by default (DEGENBOT_STATE_LOCK_DIAG), the blocked-wait warn threshold (500 ms) always on.

• bot_core/stage_telemetry.rs + bot_core/pump_telemetry.rs — BF43PM: one degenbot.epoch.run ROOT per block epoch (carrying epoch.block + epoch.seq) and one degenbot.stage.<stage> span per transition (stage.from→stage.to, queue.age_us); open rows (streaming, rewind) are force-closed past STAGE_MAX_AGE_SECS by force_close_aged (the SONJQA/G3 law). Metric series degenbot.stage.publish_cycle, degenbot.stage.rewind, degenbot.stage.rewind_duration are label-free (epoch context rides spans).

• arb_engine/engine_stages.rs — EngineStages (SZJUKL): the arb engine’s StageHandlers implementation — the dissolved coordinator/fan-out/wrapper types collapsed into one. Owns the two genuine-async-boundary channels retained from ADR-006/027: the block clock (BlockClockPipe, header ticks, never queued behind solver work) and the result batch written at the Published edge.

• arb_engine/solve_cycle.rs — the detached solve cycle (unconditional since the WFF6MM cutover; the DEGENBOT_DETACHED_SOLVES stance retired with the in-cycle arm): the solve cycle enqueues and returns, collapsing the engine-Mutex hold on the solve path to enqueue-end.

degenbot-config — typed config (KAHU5W). BotConfig is declared exactly once in schema::SCHEMA (one declaration ⇒ the typed field, the DEGENBOT_* env mapping, the TOML path, and the generated docs/rust-config-keys.md); the loader is fail-closed 12-factor (CLI > env > file > defaults) and the boot path installs it in the process-wide holder every call site reads (bot_core/stance.rs, stance::config()).

degenbot-core — block_clock_pipe.rs. BlockClockPipe / BlockNotification: the shared, engine-neutral block-clock channel (ADR-027’s direct pipe, relocated out of the retired coordinator). This live channel type is unrelated to the retired BlockClock machine.

degenbot-python — the driver shell. The PyO3 layer subscribes like any other sink: PyBot owns the pump lifecycle (PumpState, bot/pump.rs), the result-batch consumer (bot/engine/result_channel.rs) and the header block_stream hang off the Published/block-clock edges, and PySubscriberAdapter (bot/subscriber.rs) bridges PoolStateSubscriber callbacks. No raw WS stream reaches Python.

The as-built epoch cycle

1. Subscribe — WsIngestor::subscribe_with_handshake → boundary W from log-stream liveness (header-confirmed fallback past the settle window).

2. Backfill — [S+1..W] via fetch_logs applied through BotState::process_backfill_logs; the resume drops WS logs for blocks ≤ W.

3. Resume loop — per log: StageMachine::observe_log → Bot::dispatch_log (decode → write under the Streaming-confined guard → release → ledger record → notify). Per header: observe_header (a header alone NEVER advances the cursor; the D1 tombstone-by-successor rule survives unchanged).

4. Stages — on_streaming_complete (quiesce + completeness classify + debounce/early-slice gates) → on_resolve (EpochDelta::take_keys → affected paths) → on_solve → on_simulate (in-process revm, the sole executor) → on_gate (deliver/reject buckets, ADR-040) → on_publish (exactly one per quiesce cycle) → on_finalize (delivery cutoff stamped, monotone, I7).

5. Published edge — StageHandlers::on_publish is where every sink hangs: the delivery channel → Python’s result batch; the block-clock pipe → Python’s header clock; and the ADR-021 upstream RPC-disagreement verification (CompletenessDecision::Verify → assert_ws_block_complete in block_pump.rs), which is the surviving kernel of that tripwire ADR — the in-process desync-detection half was retired because write confinement makes it unrepresentable.

6. Rewind — Rewind{to_epoch} may fire from any stage (I6, at most one in flight): Epoch::rewind_to bumps seq exactly once, ReorgCoordinator runs the per-pool journal restores, the ledger is relabeled, and stale contexts fail fast.

Cheap-read posture

Writers to StateLock<RwLock<BotState>> exist only in the Streaming stage of the active epoch; Quiesced..Simulated read through cheap snapshots (spike KWKEVV). With the detached cycle unconditional (WFF6MM cutover), engine-Mutex holds on the solve path collapse to enqueue-length, so the solve is uncontended by construction — there is no drain_lock and no FIFO, and the drain_lock → engine Mutex → BotState RwLock lock-order narration of ADR-037 is historical.

Evidence

• Final-integration validation: the capture-replay regression sweep (zero divergences) and the live Jaeger soak A/B against the pre-epic operator baselines, recorded in the PLRGIN task result; the percentile replay harness lives at scripts/soak_percentiles.py.

• Spike-derived bounds behind the stage table (write-burst window, state-lock p99, rewind-restore costs): the stateview feasibility doc §3/§5.1 (docs/architecture/stateview-feasibility.md).

• Telemetry semantics (spans, metrics, force-close law): docs/telemetry-latency-playbook.md.

Legacy telemetry names (retired)

The proto-pump waterfall names are fully retired: degenbot.pump.block, degenbot.pump.log_wait, and degenbot.pump.apply_stream no longer exist as spans; the per-header beat became the degenbot.epoch.run root and the opaque children became degenbot.stage.<stage> rows. The operational lesson they encoded survives as the watchdog’s span force-close bound (LOG_WAIT_MAX_AGE_SECS / STAGE_MAX_AGE_SECS).