experiment 1 · complete · secondary external comparison
The Three-Arm Control
Same 409 questions. Same model. Three ways of handing it the record. This is the experiment that framed everything after it: where do agents actually fail on clinical records — retrieval, reasoning, or fit? Place a guess before you scroll; the reveal is the point.
what we asked
Does an agent fail on clinical records because it can’t find the data, can’t reason over it — or because the record never fits in front of the model at all?
what we ran
409 questions over ~90 real de-identified ICU patients (MIMIC-IV-on-FHIR), GPT-5.5, three arms: the raw FHIR record in context, a naive truncation (strip metadata, keep the most recent 50 per type), and a code-sandbox agent that queries the record with generated code.
what moved
Fit, not reasoning. On the 262 questions whose records overflow a 32k window, raw FHIR scores 0.0% — and when the data fits, the arms are statistically indistinguishable (truncation vs raw on fitting records: 70.1% vs 70.7% — statistically indistinguishable). Even the sandbox's pooled win is fit in disguise: on fitting records it scored 64.6% against raw FHIR's 70.7% — code is how it reaches the data past the window, not a better way to think about it (the matched-budget test). The whole staircase is one variable: whether a relevant slice reaches the window. One limit: this bundled design did not isolate every prompt, routing, and interpreter difference between the arms.
Before the reveal — on 409 real benchmark questions, how often does an agent with the raw FHIR record in context answer correctly?
Inside the 107 “fit, but wrong”: 82 answered “cannot find / truncated” and 40 asked for the earliest value the recency cap discarded — overlapping groups (≥15 in both), so they’re annotations, not segments.
lock a guess above — or keep scrolling — to reveal the results
data table
| arm | pooled (n=409) | overflow stratum (n=262) |
|---|---|---|
| raw FHIR in context (A0) | 25.4% | 0.0% |
| naive truncation (A0') | 39.4% | 22.1% |
| code-sandbox agent (A5) | 65.3% | 65.6% |
What this licenses — and what it doesn’t. This is a secondary external comparison on a weak raw-dump baseline. It frames the question; it is not a benchmark of bonfireDB, and the dashed ceiling is a heuristic bound for the overflow stratum only, never a claimed result. The program's confirmatory findings are A6a question-only selection (+9.5pp over naive truncation, replicated at +8.3pp) and QT-4 fixed microbiology vocabulary (+34.1pp on the registered stratum). The later pre-join result is exploratory: +6.8 points on the visit subset, +2.0 pooled and unresolved, with historical grading sensitivity pending.
Watch one retrieval failure unfold.
The staircase measures overflow. The next page illustrates how pagination and repeated retrieval can accumulate context; it is not a trace from this run.