Controlling logging, tracing, and telemetry¶

The operational form of the observability standard; the binding decision record is ADR-043. This is the single source of truth for what the bot emits, at what level, and how to dial it up or down.

degenbot has two first-class consumers — the pure-Rust MEV bot and the Python-driven bot — so the contract is split between the core (which emits signals and owns no sink) and the wiring (which installs sinks and chooses levels).

The four channels¶

Channel	Consumer	Default	Content
Console log	operator	on	process lifecycle and outcomes
Span (OTel)	investigator	when `telemetry.otel`	per-operation timing + structured context
Metric (Prometheus)	alerting	on	aggregates, low cardinality
Forensic file	repro	opt-in	full-field dumps

A fact is emitted on exactly one channel; it is never restated as a per-event INFO line. Any fact that paging depends on is carried by both the console and a metric, because OTel can be unavailable when it matters.

The Rust core reaches Python logging and the stderr fmt writer through one tracing subscriber (init_logging_subscriber in rust/crates/shells/degenbot-python/src/python_log_layer.rs); log:: records are bridged in by tracing_log::LogTracer. Prometheus metrics are declared in degenbot-bot/src/instruments.rs.

Levels¶

Level	Use	Budget
ERROR	abort, integrity loss, shutdown seam; mirrored by a metric	rare
WARN	degraded but continuing: tripwire, quarantine, retry exhaustion, verify failure, posture cordon, auto-recovering seam	O(events)
INFO	process lifecycle and outcomes: one block summary per block, one line per submit decision, per-phase boot/backfill/pump lines	O(1)/block + O(submits)
DEBUG	per-entity detail: per-pool, per-path, per-candidate, per-sim, per-log, phase timing, verify diagnostics	O(entities)
TRACE	per-hop/per-field dumps and raw traces	unbounded (opt-in)

The rubric is deliberately asymmetric: if a line is per-pool or per-path, it is DEBUG. The console firehose that motivated the standard was per-entity lines emitted at INFO.

Targets¶

The closed domain set is degenbot::state, degenbot::path, degenbot::solver, degenbot::sim, degenbot::pump, degenbot::exec, degenbot::verify, degenbot::ingest, degenbot::rpc, and degenbot::aave. Every diagnostic is a DEBUG event under its domain; there is no separate diag target. Engine diagnostics are emitted within an active engine span, which is what makes them visible in Jaeger.

Span names are degenbot.<area>.<verb> and metric names are degenbot.<noun>_<unit>. The console formatter derives its [area] prefix from the target, so there is no hand-written tag to keep in sync.

The control surface¶

Knob	Class	Shape
`telemetry.log_level`	behavior	closed enum `off\|error\|warn\|info\|debug\|trace`
`telemetry.diag`	verbosity	validated map `{ domain = "level" }`, ships empty
`telemetry.forensic`	behavior + sink	one capped, rotating file target
`telemetry.otel`, `telemetry.metrics_addr`	behavior	driver wiring

telemetry.diag is the console escalation knob: it raises named domains on the console without touching the OTel side (which already carries degenbot at debug). Keys are validated at config load against the closed domain set, so a typo is a boot error naming the offending key and the valid set — never a silent no-op. The map is validated even when RUST_LOG overrides it, with one WARN that it is being ignored.

Precedence¶

Precedence is a branch, not an ordering:

If RUST_LOG is present it is used as-is on every sink, the config log knobs are ignored, and the active source is named once at startup.
Otherwise telemetry.log_level + telemetry.diag compile into one EnvFilter (the console filter).
The OTel spans layer is an independent fixed default, warn,degenbot=debug, whenever telemetry.otel is on.

RUST_LOG is directives of the form target=level, comma-separated, applied most-specific-first; see tracing-subscriber’s EnvFilter docs.

Operator recipes¶

I want to …	Do this
Quietest useful run (warn/error only)	`RUST_LOG=warn ./run_bot.sh`
Default operator posture	`./run_bot.sh`
Keep `info` but raise one domain on the console	`[telemetry.diag] sim = "debug"` (or the env equivalent)
Full Rust + Python debug	`RUST_LOG=debug DEGENBOT_DEBUG=1` then the bot
Just the Rust core at debug	`RUST_LOG=info,degenbot=debug`
Full diagnostic stream in Jaeger only	leave the defaults — the OTel layer already runs `degenbot=debug`
Prometheus scrape	`curl http://127.0.0.1:9464/metrics` (override `telemetry.metrics_addr`)
Jaeger traces	see the bot-telemetry skill

The console writer¶

Exactly one console-emitting writer per process (ADR-043 §6). In the Python driver the binding is present, so the Rust→Python bridge owned by logging.py writes the console and the fmt layer is routed to io::sink(); in a standalone Rust binary the fmt layer owns it. A record is never written twice — the owner is derived from binding-present, not switched by an env var.

Every console line is LEVEL [area] message, where the area is derived from the record’s logger name — the closed domain target for bridged Rust records (degenbot::solver → solver), the owning segment for a Python-side record. The message itself carries no [tag]: a tag would state the routing decision a second time and drift from it (ADR-043 §7, enforced by the message sweep in observability_naming.rs).

The bridge’s queue is bounded (8192 records): a stalled TTY or full pipe cannot stall the per-log pump. A push past the ceiling drops the record and counts it as degenbot.log_dropped_total{sink="console"}, so the ceiling is visible in Prometheus rather than silent.

Retired verbosity flags¶

The former per-diagnostic DEGENBOT_* env flags are retired (no aliases); each maps to a domain and is now a DEBUG event under it. A boot-time WARN scans the process environment for any retired name and prints the equivalent domain. The mapping and the disposition of the four default-ON streams (verify_dbg, v2_calc_trace, dump_call_trace, sim_log_reverted_swaps) live in ADR-043 §5.

DEGENBOT_LOG_FMT is retired with them: the console owner is now derived, so the two-tunnel switch has no equivalent knob. Setting it warns at boot.

Implementation status¶

The standard is being migrated in five phases (ergo epic RAYW7I). Until a phase lands, its knob is not yet live; the tables above describe the target contract and this section tracks the gap.

Phase 1 (docs + ADR) — landed.
Phase 2 — telemetry facade + closed domain targets landed.
Phase 3 — landed. Every verbosity key in ADR-043 §5 is retired; the probes are unconditional DEBUG (or TRACE for the forensic dumps) events on their domain, and the per-entity INFO firehose is demoted. Two keys were reclassified from the §5 retire table into the behavior-flag list: state_lock.trace and state_lock.diag gate diagnostic collection cost, not log emission, so they keep their boolean config (the ADR table is corrected accordingly).
Phase 4 — landed: telemetry.log_level (closed enum, defaults to the wiring default) and telemetry.diag (validated map, ships empty) resolve both record-layer filters through degenbot_bot::telemetry::resolve_filters; an explicit RUST_LOG is a branch that wins verbatim on every sink (the config knobs are ignored, with one WARN naming the active source). The console writer is single-owner and bounded with its drop counter; DEGENBOT_LOG_FMT is retired (boot-time WARN). telemetry.forensic remains the one unimplemented knob of this phase.
Phase 5 — landed: the naming gate (degenbot-bot/tests/ observability_naming.rs sweeps production sources; spans are degenbot.<area>.<verb>, metrics degenbot.<area>.<noun> — the block-epoch root span is now degenbot.epoch.run), the message sweep (no [area] tag in a production message; the console _AreaFormatter derives it from the logger name instead), the metric-cardinality gate (metric_cardinality.rs: label names are a closed reviewed allowlist and unbounded-looking names fail outright), and the degenbot.metric_series self-metric + DegenbotMetricCardinalityHigh alert. Tracked successors for the rest: ZJUEXH (behavioral volume gate), RL7X4C (golden snapshots for boot / one block / one revert) — both landed; every [area] message tag is deleted (147 in src, plus tests/examples) and the console prefix is derived (4QYTPH).