Grade the Graders
Our first gpt-5-mini judge setup rejected 43 correct numeric answers. Later panels scored 98–99%, but they received more context and tolerance guidance. This is an audit warning, not a fair model leaderboard. Try three synthetic examples of the observed failure pattern below.
| Judge | Accuracy | False negatives (rejected a right answer) | False positives (passed a wrong one) |
|---|---|---|---|
| deterministic truth — 100% by definition | 100.0% | 0 | 0 |
| codex / GPT panel (3-vote) | 99.1% | 1 | 0 |
| Claude panel (3-vote) | 98.2% | 1 | 1 |
| gpt-5-mini (our run of the benchmark judge prompt) | 61.3% | 43 | 0 |
The first judge setup produced 43 false negatives and 0 false positives. The later panels scored 98–99%, but the conditions differed, so the table should not be read as a model ranking.
The headline number moved three times under adversarial review. Stratify the same answers correctly — or swap the judge — and "code interpreter wins big" becomes "no significant difference." That's how a benchmark lies without anyone faking a number.
Three answers an agent gave. Mark each correct or wrong — then see what the cheap LLM judge said. (Illustrative of the measured false-negative pattern; synthetic, not the DUA-restricted source questions.)
Method & honesty
Numbers come from a re-grade of the 409-question MIMIC-IV-on-FHIR comparison. The first gpt-5-mini judge setup omitted the question; later panels received the question and tolerance guidance. Accuracy is checked against deterministic numeric truth across 111 arm-answers. The three interactive examples are synthetic analogues because we cannot redistribute MIMIC. Bonfire’s own deterministic Synthea benchmark is planned work and has not produced a product result. See the benchmark →
Use deterministic grading where the answer permits it.
Bonfire’s planned benchmark will assert database and FHIR state directly. It has not yet produced a product-level result.