When an agent gets a clinical answer wrong, is it the reasoning — or the data we gave it?
Often it is the evidence interface, not just the reasoning. Here is what held up, what failed, and how much to trust each result. The full write-up has the numbers behind it.
A real chart can overflow the model's context window.
Hand an agent the raw record and, on a full chart, it spills past the context window — how much the model can read at once — so the agent never sees the answer. Hand it the few records the question needs, and the same model answers correctly more often.
Same chart, same model. Whole dump: ~57% right. The right kinds of record: ~73%. An oracle condition handed over the exact records and reached ~95%; a real system still has to find them. Exploratory, synthetic charts, 40 questions.
Three ways to hand over a chart.
Raw FHIR dumped in; a blunt cut that ignores the question; a code sandbox that crunches the records outside the prompt. The sandbox scored higher, but that historical arm bundled code execution, prompts, and routing, so it does not isolate one cause.
Then a test we called in advance: picking for the question beat a frozen blunt-cut baseline +9.5 points (54.3% against 44.7%, a separate paired run, not the 39% bar above), on 43% less data — that comparison was preregistered. An exploratory follow-up added a deterministic pre-join on top: +6.8 points on the questions that name a specific visit, and nothing where no join is needed; pooled over all 409 questions its gain is +2.0 points and not significant. Its protocol was written before the run but never Git-anchored, so it stays exploratory.
What held up, what did not, and what remains open.
The tests do not support one universal packet. They support a sequence: select for the question, translate the chart’s vocabulary, and traverse only when the task requires a path.
Can an agent find the right data itself?
No. It guesses codes and misses — fine on vitals, bad on diagnoses — and a stronger model barely helps (both fail badly). It needs the chart's real codes handed to it.
mechanism · fake charts, smallDoes a focused slice help?
A lot. The answer buried in a full chart is right ~57% of the time; sent the right kinds of record, ~73%; an oracle condition handed over the exact records and reached ~95%, but a real system still has to find them.
explored · ~40 questionsWhat's the cheapest way to make that slice?
A cheap model picks the records, a strong model reads them. That beats throwing the strong model at the whole chart — for less money.
explored · ~30 questionsWhen does traversal help?
Only when the task needs a path. Turn-by-turn walking performed poorly in an early star-shaped chart probe, and QT-4 did not promote traversal beyond vocabulary. A11 then built a separate task where the answer sat at the end of a path: bounded traversal scored 119/120 against 24/120 for the vocabulary-only packet and recovered terminal evidence on 96/96 answerable cases.
bounded mechanism · constructed path taskA11b tried to isolate event grouping beyond the same traversed evidence. Its registered normalized labels tied at 288/384 in every arm, but forensic review found that normalization hid a real insufficiency signal. We kept the strict artifact, withdrew the behavioral null, promoted nothing, and designed a fresh successor test.
no promotion · successor reseal requiredSeparately, we tested the join on the real charts. 176 of the 409 questions — 43% — name a specific hospital or ICU visit, so answering means a join: connecting events to the right visit. A deterministic pre-join finds that visit and scopes the packet to it. On those 176 questions:
That gain matched a protocol written before the run (p=0.0075) — but the protocol was local-only and never Git-anchored, so it stays exploratory. Pooled over all 409 it's +2.0 points, p=0.26 — not significant, because most questions don't need a join. And where the join fires, the packet shrinks from a median 35k characters to 20k.
exploratory · grading sensitivity pendingOne follow-up asked whether the join has to be pre-computed. We gave the agent tools carrying the same join semantics and re-ran the 176 questions:
No accuracy difference was detected, and equivalence was not established: +4.0 points with a 95% interval from −8.7 to +17.6. The agent used 4.061× the cumulative input tokens. Historical arm-label exposure makes this exploratory.
exploratory · 176 questionsIs it safe?
Real work. A wrong slice gives confident wrong answers, and fake data slipped into the slice is trusted — so being correct and cited isn't optional.
mechanism · fake chartsDoes "fit, not cleverness" hold on real data?
On 409 real-chart questions: raw FHIR 25%, blunt cut 39%, sandbox 65% — and the sandbox's pooled edge tracks the overflow stratum, but the arm bundled several mechanisms. This is a secondary comparison, not a causal proof.
measured · 409 questionsThe models get benchmarked all the time. The data layer between the chart and the agent doesn't — and that's the layer an app actually owns.