the null levers · complete · the graveyard
The Null Levers
We pulled every lever everyone recommends. The meters didn’t move. Publishing the nulls is the point: knowing which fixes don’t work is half the finding — and the reason the one that did is worth trusting. Place your bets before you pull.
what we asked
The field’s standard advice for agents on FHIR: better tools, craftier queries, more thinking, let it code. Do any of these actually move answer accuracy when you hold the budget constant?
what we ran
Matched-budget ablations over the same benchmark questions: a tool-catalog dose-response staircase (1 → 8 tools), payload-shaping coaching, reasoning-effort steps, a code-interpreter arm, reference-expansion tooling, and retrieval-quality metrics against gold resources.
what moved
Nothing on this wall. Every confidence interval crosses zero — and the one big-looking effect (+39.9pp pooled for code) decomposed entirely into overflow-avoidance, vanishing at matched budget. The lever that moved lives at the bottom, and it isn’t compute: it’s what reaches the window.
Six interventions everyone recommends. Which of them would you bet moved agent accuracy? Pick any, then pull.
place your bets and pull — or keep scrolling — to reveal
"But the code sandbox won the three-arm control." It did, pooled: 65.3% against 25.4% for raw FHIR. The two results describe two different jobs. In the three-arm control, code is a data-access strategy: the sandbox queries the record programmatically, so the chart never has to fit the context window, and on the overflow questions it scores 65.6% where the raw in-context arm scores exactly 0% (blunt truncation reaches only 22.1%). That entire pooled win is overflow-avoidance. On the questions where the chart did fit, the sandbox scored 64.6% against raw FHIR's 70.7%: no better. What this page tests is code's other job, a reasoning lever over data the model can already see, at matched budget. That job is the one that does nothing. Code that fetches less, works; code that thinks harder, doesn't.
What this licenses — and what it doesn’t. These are exploratory ablations, several underpowered by design (n=25–30 per cell; the tool-catalog run’s minimum detectable effect was ~34–46pp — its enormous CI on the meter is the honest picture). Two levers trace to a destroyed, unreproduced Opus run and are badged internal. The licensed reading is directional: within matched budget, none of these interventions demonstrated an accuracy effect — while the selection result below them is the program’s first preregistered confirmatory finding.
Six levers that didn’t move. One that did.
Selection isn’t a model trick — it’s a data-layer responsibility. Selection is the program’s strongest replicated data-layer result; Bonfire itself remains untested.