the selection experiment · preregistered · replicated
Question-aware selection worked twice
Two paired 409-question runs found the same advantage. Hypothesis and decision rule committed before the run, planner frozen at a git tag, packets sealed with per-question hashes. The initial run improved accuracy by 9.5 points at 43% less payload. After repairing the shared time assumption and pinning the runtime, the replication improved accuracy by 8.3 points and used 35.5% fewer accepted-attempt model tokens.
what we asked
Does deterministic, question-only selection — parse the question, fetch the right resource type and date window, keep both temporal endpoints, resolve references — beat query-blind truncation of the same record?
what we ran
The same 409 questions through both arms, paired. The planner is a deterministic keyword parser — a floor, not a tuned system — provably reading only the question, patient, and assumption fields (no benchmark-construction shortcuts). Grading: deterministic where unambiguous, 3-vote panel otherwise.
what moved
Initial: 54.3% vs 44.7% (+9.5pp): of the 79 questions where exactly one arm was right, 59 vs 20 favor selection, on 43% smaller packets. In the assumption-fixed replication: 54.5% vs 46.2% (+8.3pp), using −35.5% accepted-attempt model tokens.
3/3the discordant pairs decide it
59 flipped toward selection, 20 against — net +39 of 409 = +9.5pp, McNemar p=1.3×10⁻⁵ on exactly these pairs
selection only right (59)truncation only right (20)both right (163)both wrong (167)
net +39 / 409 = +9.5pp95% CI +5.4…+13.7 (patient-cluster bootstrap) measured in this benchmark
The dot plot above shows the initial A6a run. The repaired replication is reported separately because its corrected assumptions and token receipts must not be mixed with the initial run.
measured in this benchmark
What this supports. Deterministic question-only selection beat query-blind projection twice, on this dataset, model family, and harness. It does not validate bonfireDB, cross-model generality, or graph traversal; the artifact review applied 8 label corrections and the +9.5pp was unchanged. The dot plot’s 59/20 discordant counts and 43% serialized-payload reduction belong to the initial run; the 35.5% accepted-attempt model-token reduction belongs to the repaired replication. Concordant cell counts in the figure (163/167) are derived arithmetically from the published margins.
Selection belongs below the prompt.
bonfireDB is being designed to build the bounded, cited slice before the agent runs. This experiment supports that approach; it does not yet validate the product.