MCP benchmarks are easy to lie with

"We added MCP and accuracy went up" is not a result. It is a place to start asking what changed: the tool schema, the prompt, the payload, the evidence, the model, the judge, or the budget.

The Model Context Protocol gives agents a standard way to call tools. That matters. But MCP is the transport, not the treatment. In clinical data, the treatment is the evidence returned to the agent: which records, which fields, which date window, which citations, and how much context it consumed.

The under-specified claim

A benchmark that says "MCP improved accuracy by 20%" usually leaves out the part that matters. Did MCP help, or did the tool return a smaller and cleaner payload? Did the tool description teach the model the answer shape? Did one arm overflow and the other fit? Did the judge punish terse correct answers from one arm more than another?

Those are not pedantic questions. They are the difference between a product insight and a confound. In our measured comparison (MIMIC-IV-on-FHIR, a FHIR-AgentBench fork, 409 questions), a few early stories looked like tool wins and did not survive controls. A tool-catalog lead turned out to be one arm dodging the context window, not reasoning better: raw FHIR scored 0% on questions that overflowed and 70.7% when the chart fit. A traversal tool gained +9.1 points but failed significance (p=.219, not promoted). And our own AI grader was one-sided — 43 false negatives, 0 false positives, 61.3% agreement with the known numbers — enough to move a headline until we caught it. Our internal Synthea suite grades deterministically against resulting database and FHIR state, which avoids that judge risk. Two later exploratory join tests found a promising +6.8-point visit-subset signal, but only +2.0 points pooled, and an unresolved agent-versus-pre-join comparison (+4.0 points; 95% interval −8.7 to +17.6). The agent used 4.061× the cumulative input tokens. Historical arm-label exposure and an incomplete model pin mean those join results still need a sensitivity re-grade.

The cheating surface

You can accidentally make an MCP benchmark lie without writing any malicious code. Here are the common failure modes:

VariableHow it liesGuardrail
Tool schemaA better description becomes hidden prompt coaching.Hash and publish tool schemas per arm.
Returned payloadOne arm returns less noise and avoids overflow.Report source IDs, bytes, tokens, and field selection.
Context capOverflow looks like worse reasoning.Track overflow separately from wrong answers.
Skill textA FHIR playbook may just be a longer prompt.Run neutral-length and placebo controls.
JudgeA weak grader can invert the headline.Audit against deterministic ground truth where possible.
CostAn accurate arm may be wildly inefficient.Report tokens, calls, and spend next to accuracy.

What a credible MCP benchmark freezes

A clinical MCP benchmark should publish enough artifact detail that another team can tell what the agent actually had access to. At minimum:

  • Server identity: implementation commit, startup command, MCP URL, branch, and version.
  • Tool contract: names, descriptions, JSON schemas, capability set, and schema hash.
  • Evidence returned: source resource IDs, payload bytes, token estimate, citations, and any projection rules.
  • Prompt and skill text: model instructions, answer schema, skill hash, neutral-control prompt, and placebo prompt.
  • Runtime trace: calls made, repeated-call rate, errors, overflows, timeouts, and final answer.
  • Grading: deterministic scorer where possible, LLM judge labels where necessary, judge audit, and paired statistics.

Without those artifacts, "MCP helped" is too vague. The real result might be "a better clinical read model helped," "a shorter payload helped," or "the benchmark rewarded one prompt style."

The next fair test

The next confirmatory comparison freezes the best symmetric non-graph protocol, then compares it with the same protocol plus graph-computed evidence on a fresh, patient-disjoint holdout. The previous development questions are burned. That test, distinct from the completed visit-join tests above, is preregistered and has not produced a result.

That design asks whether graph computation adds value after stronger prompting and retry behavior receive the same opportunity. It still will not validate the full Bonfire product until the governed read contract itself is the treatment.

The claim ladder

There are levels to this. Each one licenses a different sentence.

  • Smoke test: "The MCP server starts and tools/list works."
  • Exploratory pilot: "This tool surface looks promising on a small slice."
  • Artifacted benchmark: "Under frozen schema, prompt, model, grader, and evidence logs, this arm improved accuracy/cost on this benchmark."
  • Product claim: "The governed Bonfire read contract improves agent chart-reading under policy." That requires the real read contract, not just an expanded proxy.

The last sentence is the one we want to earn. The first three are how not to fool ourselves on the way there.

What to report next to accuracy

Accuracy alone is too easy to game. Every comparison should include retrieval precision and recall where possible, source IDs, payload bytes, prompt and completion tokens, LLM calls, repeated-call rate, failure taxonomy, and cost. If an arm wins by sending twice as much evidence or spending three times as much, that is useful—but it is a different product decision.

Keep reading

Sources

Benchmark the read layer, not the vibes.

bonfireDB is building MCP-facing clinical context tools with source IDs, citations, policy scope, and reproducible benchmark artifacts.

Become a design partner →Agent context tools