the lab notebook · updated 2026-07
The Experiments
Selected experiment reports. Each card states the current evidence status and the narrow claim it supports. W1 and W2 are exploratory because their historical panel exposed arm labels and did not retain a complete model pin.
Three-arm control
Raw FHIR vs blunt projection vs a code sandbox — where do agents actually fail?
supports: Secondary external comparison on a weak raw-dump baseline — frames the question; not a benchmark of bonfireDB.
A6a — question-only selection
Does deterministic question-only selection beat query-blind projection?
supports: Two paired 409-question runs support the same narrow claim: deterministic question-only selection beats query-blind projection.
Grade the graders
Can a single LLM judge be trusted to score clinical answers?
supports: One judge setup rejected many exact numeric answers. Later panels were more accurate under different conditions, so the contrast is an audit warning, not a model leaderboard.
The null levers
Do tool catalogs, payload coaching, thinking time, or code fix it?
supports: None of these small matched-budget tests demonstrated an accuracy gain; several were underpowered or unreproduced.
QT-1 to QT-3: three clean nulls
Do pinned references, aggregate summaries, or reserved endpoint fields improve the A6a packet?
supports: No. All three paired 409-question tests stayed within about a quarter point of A6a. More preserved data and more summary structure did not improve accuracy, so none was promoted.
W1a: the pre-join test
43% of the questions name a specific visit. Can a deterministic join, computed before the model sees the record, improve accuracy on them?
supports: Exploratory support: the visit-specific subset moved +6.8 points on 176 questions. The pooled +2.0-point result was unresolved, and no gain was detected on patient-scope questions. Historical arm-label exposure and a missing model pin require a sensitivity re-grade.
W2a: who should drive the join?
Given tools that can express the join, does an agent match the deterministic pre-join?
supports: No accuracy difference was detected, and equivalence was not established: 72.2% versus 68.2%, with a wide interval (−8.7 to +17.6 points; p=.41). The agent used 4.061× the cumulative input tokens. The 53 discordant questions motivate a hybrid development test.
QT-4: can the agent name what it needs?
FHIR fetches by code, and agents guess diagnosis codes wrong two times in three. Does handing over the chart’s own vocabulary fix it?
supports: The chart vocabulary helped where code guessing failed. An exploratory synthetic probe found 33% exact SNOMED guesses and a +6.6-point dictionary lift. QT-4 then confirmed fixed microbiology vocabulary on an untouched holdout (10/44 → 25/44, +34.1 points, p=0.000275) without harming the 330 controls. Traversal was not promoted.
A11: when traversal is actually required
If the answer sits at the end of a path, can bounded traversal recover it?
supports: Bounded traversal recovered terminal evidence on all 96 answerable constructed path questions, versus 0/96 for the vocabulary-only star packet — which abstained on every question, so its 24 correct answers are all correct abstentions. This supports the mechanism on a deliberately path-required task, not graph storage or natural-chart generality.
A11b: the result that forced a redesign
Does typed event grouping add value beyond the same traversed evidence and deterministic aids?
supports: The registered normalized labels tied at 288/384 in every arm. Forensic review found that normalization hid a real insufficiency signal, so the behavioral null was superseded. No feature was promoted; a fresh successor test must be resealed before it runs.
Is it safe? (the safety experiments)
The slice can be wrong, poisoned, or missing. What happens then, and what defends?
supports: Exploratory only (synthetic charts, small n, one model, one run each): a wrong slice collapses 86.7% to 16.7% with confident wrong answers; crude injection was obeyed 0%, but plausible poison was trusted 100%, and a cheap auditor recovered 79.2%. This supports a design direction—provenance, audit passes, and typed ingestion—not a production safety claim.
Does the result travel?
Do selection, vocabulary, and bounded traversal still help with other models, FHIR servers, and patient-disjoint test sets?
status: No result yet. The first model-and-effort grid used the wrong denominator, so it is excluded from every generality claim. The replacement must freeze the same packets and scoring contract across cells.
A7: does governance preserve usefulness?
Can an agent get the same evidence through cited, policy-checked handles without arbitrary chart access?
status: No result yet. This must hold evidence constant and measure correctness, disclosure, denial handling, latency, and cost separately.
A12: can honest errors help?
When a query is malformed or unsupported, can a precise error help the agent recover without leaking data?
status: No result yet. The test must separate strict rejection from useful correction and count both recovery and unsafe disclosure.
A14: does identity change the right answer?
What should the same request return for a patient, clinician, backend service, or unauthorized caller?
status: No result yet. A correct denial is not an accuracy failure; the benchmark must score authorized evidence, valid refusals, and disclosure separately.
The claims stop where the evidence stops.
This notebook records the wins, the nulls, the broken tests, and what must run next. Bonfire’s own product benchmark is still open.