experiment 7 · the vocabulary arc · exploratory probes + QT-4 measured · complete
Can the Agent Name What It Needs?
In FHIR you fetch by code, and agents guess diagnosis codes wrong two times in three. This page is the whole arc: the probe that found the gap, the stronger model that didn't fix it, the oracle trap we caught in our own test, the error-feedback patch, and the pre-registered holdout confirmation (QT-4) that made vocabulary the first mechanism in the program to survive a measured test.
what we asked
Before an agent can answer, it has to fetch, and FHIR fetches by code. If the agent's chosen code isn't the one this chart uses, the search comes back empty and the answer is wrong. So: can the agent name the codes a chart actually uses, and if not, is that a reasoning gap or a knowledge gap? The framing lives in the research write-up (Q1).
what we ran
Five tests, weakest first. Four exploratory probes on synthetic (Synthea) charts: a guess-rate census by code system, a model-strength ladder, an answer-lift test we had to de-oracle and rerun, and an error-feedback patch. Then QT-4, the pre-registered measured confirmation: fixed microbiology vocabulary evaluated on a fresh 374-question untouched holdout over the real ICU charts behind the benchmark.
what moved
The gap is real and it is knowledge, not reasoning: SNOMED guesses land 33% of the time, a flagship model only reaches 30% blind, and handing over the chart's own dictionary takes the same questions to 100% (a ceiling). Honestly de-oracled, the delivered lift is +6.6 points. QT-4 then confirmed it where it counts: the registered stratum moved 10/44 to 25/44 (+34.1 points, p=0.000275, promoted) while 330 negative controls held still.
diagnosis concepts have many valid codes; a chart uses only a subset. guess the wrong one and the search returns empty.
a stronger model doesn't fix it exploratory · n=10 SNOMED condition questions
The flagship was slightly better than the smaller model, and both fail badly: 30% vs 20% is not a fix, it is the same failure at a higher price. The dictionary arm at 100% is near-oracle, a ceiling: it says the failures are a knowledge gap the chart's own vocabulary can close, not what a shipped system delivers.
the oracle trap we caught in ourselves exploratory · the methods lesson
the dictionary knew the answers’ codes; this flatters the mechanism
+6.6 points, real but far smaller than the number that flattered us
Our first answer-lift test handed the agent a dictionary built from the gold answers' own codes: an oracle wearing a treatment's clothes, and it printed a flattering 25.8-point gap. Rebuilt from the chart alone, the honest lift is +6.6 points, real but far smaller. The first number flattered the mechanism; the honest number survived. That reflex, hunt your own confounds before publishing, is the same one that later caught the overflow trap and the grader trap.
error feedback helps recovery exploratory · n=24
When a search with a wrong code returns empty, silence carries no information: the agent can't tell "nothing exists" from "wrong name". Surfacing "that code matched nothing; these code systems are present in this chart" moved recovery from 75.0% to 83.3%, +8 points, a cheap second line of defense behind the dictionary.
+34.1 points on the registered stratum · p=0.000275 · promoted
The synthetic probes predicted a knowledge gap; QT-4 is that prediction surviving contact with real charts, a registered stratum, and an untouched holdout. The chart-scoped dictionary is the cheapest mechanism in the program and the first to survive a pre-registered measured test.
What this licenses. Agents cannot reliably name the codes a chart uses, and it is a knowledge gap, not a reasoning gap: hand over the chart's own vocabulary and the failures largely disappear. QT-4 licenses the measured claim for its registered microbiology stratum only: +34.1 points on 44 questions, controls clean, tokens down 13.9%. Never quote the 100% dictionary arm or the 93.5% oracle-flavored number as delivered accuracy; both are ceilings, and the honest exploratory lift is +6.6 points. Traversal earned nothing here: 29/44 was not resolved against vocabulary alone, so only vocabulary was promoted. The join mechanisms live on the pre-join test and who should drive the join; the full narrative is in the research write-up.
The dictionary closes the naming gap. The join closes the linking gap.
Vocabulary tells the agent what things are called; it still has to connect them to the right visit. W1 and W2 explore that join question, but their historical grading exposed arm labels and still needs an opaque-ID sensitivity re-grade.