ADR-066: Deterministic Stub Generation — experimental-inspect + pyo3-introspection retire the hand-maintained .pyi set

Status: accepted (architecture). Decided at the architecture-review grilling. The stub-diff spike is the first implementation gate and reopens this decision only if generation proves unable to cover the seam surface.

Context

ADR-053 evaluated pyo3-stub-gen — a proc-macro rewriter of the seam’s source — rejected it, and kept the 27 hand-maintained stub files under src/degenbot/_ffi/, policed by mypy.stubtest (tests/ffi/stubtest_allowlist.txt) and a bespoke AST drift gate. The ecosystem has since moved: PyO3’s experimental-inspect feature (documented at pyo3.rs v0.29.2) embeds introspection data in the built cdylib, and the first-party pyo3-introspection crate (0.29.2, released in lockstep with the workspace’s pyo3 ^0.29) reads it back — introspect_cdylib(path) returns the module model and module_stub_files() emits the root __init__.pyi plus per-submodule stubs: exactly the artifact shape the seam hand-maintains today.

The governing principle from the same design session: wherever deterministic generation is available, use it. A Rust-only consumer and a Python driver must reach the core on equal footing, and the PyO3 layer stays minimal — hand-written seam artifacts are a third place for drift to hide.

Decision

D1 — A named stub-generation lane. A build of degenbot_rs with pyo3’s experimental-inspect feature exists solely to be introspected; a just recipe runs introspect_cdylib over that build and writes the stub files. Release wheels exclude the feature, per the release-feature policy.

D2 — Generated stubs are committed, with a regenerate-and-diff gate. The stub set stays in the tree (Python imports need it beside the compiled module), but source and stubs may only agree: the REGEN_*-pattern drift gate already used for docs/rust-config-keys.md fails on any diff.

D3 — The existing gates become generator verification. mypy.stubtest against the running extension and the AST drift gate stay in place, re-read as checks on the generator’s output. Generator gaps live in tests/ffi/stubtest_allowlist.txt and only there — nobody hand-edits a generated stub on top, because that re-enters the drift blind spot this ADR closes.

D4 — The principle spans the seam. The typed configuration surface owed to ADR-065 (a named-property projection over the verdict, replacing dotted-path string lookups) is emitted from the config_schema! declaration site rather than hand-written, and the dead seam doors beside it (verification_retry_policy_defaults, the duplicate discovery_batch_size clamp) close. ADR-053’s pyo3-stub-gen rejection stands; its “the hand-maintained stub set stays” decision is superseded.

Consequences

  • The first implementation step is the spike the decision names: generate against today’s cdylib and diff the output over the 27 hand-maintained files. That delta is precisely the future contents of the allowlist — and the honest measure of how far annotation generation still is (PyO3 calls it “a first step”; issue #2454 tracks the rest).

  • pyo3-introspection tracks pyo3 in version lockstep, so the generator pin follows the workspace’s pyo3 pin; a pyo3 bump implies a generator bump and a fresh diff.

  • The pure-Rust consumer is untouched: stubs are a Python-visible artifact only.

Alternatives considered

  • Keep hand-writing (ADR-053’s path). Rejected: the drift gates exist to police human transcription work a deterministic tool now performs.

  • pyo3-stub-gen. Stays rejected per ADR-053 — it rewrites the seam’s source via proc macros instead of introspecting the built artifact; it was never this proposal.

Gate outcome — the first spike

The stub-diff spike measured the mechanism against the real seam: the generator (pyo3-introspection 0.29.2, audited) works, introspection data embeds correctly, but pyo3-macros-backend’s fn-based #[pymodule] expansion passes empty member lists and the incomplete flag, and this seam registers imperatively (PyModule::new + add_function + add_submodule) at ~27 sites. Result: 27 of 27 deltas are generator-gap at the module-structure level; zero generator-correct, zero generator-broken.

The decision therefore stands, with a named prerequisite now visible: the registration surface must move to declarative #[pymodule] mod form (tracked as an implementation task). The post-conversion re-gate is folded into that task’s acceptance: if generation still cannot cover the seam after conversion, this ADR reopens under its own terms.

Post-conversion re-gate: the decision stands. The root module introspects 28 classes, 36 functions, and all 26 declarative submodule trees with reachable members — 27 generated stubs, 1:1 with the hand tree, incomplete=false on 24 of 26. Classification: generator-correct in bulk (including catching ten runtime members of degenbot._ffi.simulation missing from the hand .pyi, soaked today by the stubtest allowlist), generator-gap confined to the create_exception! island types, imperative U256 attributes, the #2454 annotation frontier, __all__ emission, and __new__-vs-__init__ representation; generator-broken: zero. The stub-generation lane builds with the gap list as the allowlist’s origin.