experiment 5 · W1a · exploratory · complete · grading sensitivity pending
The Pre-Join Test
43% of the benchmark names a specific visit, and those questions often require a join. The observed visit-specific gain is promising. The pooled result was unresolved, and the historical grading path exposed arm labels. This page reports the numbers and the limits together.
what we asked
176 of the 409 measured questions (43.0%) name a specific hospital or ICU visit: “your last hospital visit”, “the current admission”. Answering them means connecting events to the right visit. Can a deterministic engine do that join before the model ever sees the record, and beat the frozen A6a selection packet?
what we ran
A deterministic pre-join engine: detect the visit reference in the question, order the patient’s visits, select the intended visit, then scope retrieval to it. Both arms ran contemporaneously. Deterministic grading covered 268 arm-answers; a 3-vote same-family panel covered 550. That panel was not arm-blind: its item IDs exposed arm names. The answer model and reasoning effort were not retained.
what moved
The pooled result was unresolved: 56.0% vs 54.0%, +2.0 points, p=0.256. On the 176 visit-specific questions, the observed result was 68.2% vs 61.4% (+6.8 points, p=0.0075, flips +15/−3). On 226 patient-scope questions, no gain was detected: −1.8 points, p=0.50.
+2.0pp · McNemar p=0.256 · 95% CI [−0.9, +4.5]pp · flips +23/−15
+6.8pp · McNemar p=0.0075 · 95% CI [+1.8, +12.4]pp · flips +15/−3
−1.8pp · p=0.50 · 95% CI [−5.8, +2.6]pp
Where the join fires, the packet is smaller and better: median 35k chars down to 20k (−43%), and −36% payload across the whole run. Reproducibility check: the contemporaneous A6a rerun scored 54.0% against 54.3% eleven days earlier.
the four data traps the join had to survive
Encounters arrive as nested families: ED, ICU, and ward encounters inside one admission. Counting them raw over-counts a patient’s visits by roughly 3x, so “your last hospital visit” selects the wrong one. The builder joins each family into one visit before ordering.
MedicationRequests can point at encounter references that never resolve. A pure reference join silently drops those events, which is why the frozen engine pairs the reference join with a date-window fallback.
Roughly a third of lab observations carry no encounter reference at all. Scope retrieval by reference alone and they vanish from the visit; the date-window fallback is what keeps them in the packet.
ED-only encounters look like hospital stays but are not admissions. The identifier systems discriminate between the two, so visit selection has to be facility-aware or it answers about the wrong kind of visit.
The builder went v1 to v4 on the dev split. Its preserved hash is 6fd13dd5…. The local protocol file predates the answer run, but it was not committed before the run and was modified after results existed. That limits what the word “preregistered” can honestly mean here.
What this supports. A deterministic visit-to-resource join is worth carrying into the next experiment: the observed gain concentrated in visit-specific questions and used less context. It does not license +6.8 points as an overall gain, prove zero effect elsewhere, prove a graph database is needed, or survive the historical grading flaw by itself. The full limitations are in the commit-pinned result receipt.
Next question: should the data layer or the agent perform the join?
W2a gives the agent join-capable tools. Its result is useful, but unresolved.