The evidence packet is the product

The useful thing is not a bigger prompt, a fancier tool list, or a sandbox by default. It is the clinical read object: bounded, cited, policy-scoped evidence that says exactly what the model saw.

The core Bonfire lesson from the benchmark is smaller and sharper than "agents need tools." Agents need the right clinical evidence, in a shape that fits the question, before the model starts reasoning. That shape is an evidence packet: a query-aware, cited, replayable clinical read.

The wrong abstraction is "give the agent FHIR"

FHIR is the right interoperability substrate, but raw FHIR is not a good agent interface. A real chart is a graph of resources, references, dates, codes, narrative text, and repeated measurements. If the agent pulls a raw bundle into context, two bad things happen: the useful evidence gets buried in noise, or the record simply does not fit.

We are building a Bonfire-specific Synthea benchmark with deterministic DB/FHIR state grading and a zero-leak security gate; it has not produced a product-level result. The measured evidence comes from a separate MIMIC-IV-on-FHIR FHIR-AgentBench fork. At a fixed 32k context budget, raw FHIR scored 0/262 when records exceeded the window. When records fit, raw FHIR and blunt projection were statistically indistinguishable; the code arm was lower on that stratum. The bundled experiment did not isolate every prompt, routing, and interpreter difference.

What an evidence packet is

An evidence packet is the thing the model is allowed to read. It is not a transcript of every tool call. It is not a compressed FHIR dump. It is the clinical read result the data layer can defend later.

{
  "packet_id": "sha256:...",
  "question_id": "1697a8...",
  "intent": {
    "patient": "10021487",
    "resource_types": ["Observation"],
    "date_window": { "since": "2116-03" },
    "temporal_operator": "first"
  },
  "source_queries": [
    "/Observation?patient=...&code=...&date=ge2116-03"
  ],
  "evidence": [
    {
      "source_id": "Observation/ba2f7084-...",
      "effective_time": "2116-03-18T09:15:00Z",
      "display": "microbiology organism",
      "value": "BACTEROIDES FRAGILIS GROUP",
      "citation": "Observation/ba2f7084-..."
    }
  ],
  "insufficiency": null
}

The exact schema will change. The invariants should not: resource intent, patient scope, date window, first/latest preservation, source IDs, citations, freshness, packet hash, and a structured insufficiency result when the evidence is not enough.

Why this is a product object, not prompt glue

A prompt disappears after the model call. A packet can be replayed. That is the difference between "the model probably saw the chart" and "here is the exact evidence returned to this agent, under this scope, at this time, from these records."

  • For accuracy: the packet keeps the clinical facts the question asks for, such as first, latest, min, max, date-windowed values, and reference-resolved medications.
  • For cost: it reports token size, resource count, repeated-call waste, and whether selection actually saved context.
  • For safety: it is the one place to enforce patient, tenant, consent, purpose, and minimum-necessary boundaries before PHI reaches a model.
  • For audit: it gives the source IDs and packet hash needed to reconstruct the read later, alongside AuditEvent and Provenance records.
  • For product: it becomes the stable contract that SDK calls, MCP tools, hybrid search, and future operators can all return.

A6a measured selection. QT-4 confirmed vocabulary. A7 must test the governed product.

The honest status: A6a is a measured research win for deterministic question-only selection over its frozen query-blind control. QT-4 then confirmed fixed microbiology vocabulary on a fresh 374-question valid-split holdout. On the registered 44-question stratum, vocabulary moved 10/44 to 25/44 (+34.1 points, p=.000275) while the 330 negative controls stayed inside the safety bound. Neither is a Bonfire product result. QT-4's bounded-traversal arm reached 29/44, but its incremental +9.1-point contrast remained unresolved (p=.219) and was not promoted. A7 must still show that the governed read-layer contract adds value as a product.

  • A6a: query-aware in-context packets — replicated. The initial 409-question run improved accuracy by 9.5 points. The assumption-fixed replication improved accuracy by 8.3 points and used 35.5% fewer accepted-attempt model tokens.
  • QT-4 valid374: terminology vocabulary — holdout-confirmed. Vocabulary scored 25/44 versus 10/44 for A6a-r (+34.1 points, 95% CI +17.9 to +50.0, p=.000275). Vocabulary plus traversal scored 29/44, but traversal's +9.1-point increment was unresolved (p=.219) and not promoted.
  • A7: governed read-layer proxy. Start from A6, then add deterministic reference resolution, terminology summaries, source citations, read-contract shape, and insufficiency metadata. Still a proxy until policy gates, audit trail, capability negotiation, and explicit denial semantics are product-real.

The right table for follow-up work is not just accuracy. It should show overflow rate, packet tokens, source-resource count, citation support, cost per question, and pre-declared paired comparisons. Otherwise we are just creating a prettier version of the same benchmark trap.

What this does not claim

The packet framing does not say that FHIR is bad, that MCP is irrelevant, or that sandboxes are useless. It says those are implementation surfaces. The load-bearing object is the governed clinical read.

  • Do not claim Bonfire has already beaten the sandbox. That is what A6/A7 must test.
  • Do not claim projection solves the problem by itself. The measured projection was query-blind and failed many first/earliest questions.
  • Do not claim a sandbox is required. The current evidence says a sandbox is one way to avoid overflow, not the unique mechanism.
  • Do not claim QT-4 proves graph traversal or graph storage is better. Vocabulary passed untouched-holdout promotion; the traversal increment did not.
  • Do not claim an MCP server improves accuracy unless the returned evidence, payload size, prompt, model, judge, and cost are frozen and reported.

Where this fits in Bonfire

Bonfire is not trying to make raw FHIR a little easier to prompt over. The product bet is that a clinical backend should compile typed operational data into the evidence an agent needs: scoped, fresh, cited, and small enough to reason over.

That is why the roadmap moves from query-aware packets to governed read tools, then to structured operators, hybrid memory for notes, cohort aggregation under ABAC, and graph/timeline retrieval. Those are all different ways of producing better evidence packets.

Keep reading

Sources

Agents need evidence, not dumps.

bonfireDB is building the clinical data layer that returns bounded, cited context to agents before they reason over the chart.

Become a design partner →See the benchmark