the meta-experiment · complete · why every other number is trustworthy
The Judge Re-Grade
Before trusting the scores, we scored the scorers. A single-LLM judge was 61.3% accurate against deterministic ground truth — and the answers it rejected were exactly right. This experiment is the reason the benchmark numbers here are graded deterministically where a known answer exists, with multi-vote panels for the rest.
what we asked
The field’s default is one LLM judging another’s answers. Against non-negotiable ground truth — 111 numeric answers checkable by arithmetic — how accurate is that judge, and in which direction does it err?
what we ran
We checked the first gpt-5-mini judge setup against deterministic numeric truth, then ran later 3-vote panels as sensitivity checks. The setups were not symmetric: the first judge omitted the question, while the panels received the question and tolerance guidance. The panels landed at 98.2% and 99.1%; the first setup sat at 61.3%.
what moved
The single judge failed in one direction only: 43 false negatives, zero false positives — it rejects exact-right answers. A bias like that doesn’t add noise; it subtracts from whichever arm you’re testing, which is how it inverted our code-interpreter headline’s sign.
The kicker: the 43 answers the single judge rejected had median relative error 0.0 — they were numerically exact. It wasn’t grading accuracy; it was punishing format.
What this supports. This first judge setup was not reliable enough for confirmatory grading; its error was directional, large, and sign-inverting. Because the later panels received more context and tolerance guidance, the 61.3% versus 98–99% contrast is an audit warning, not a fair comparison between judge models. Current policy is deterministic grading where arithmetic decides and multi-vote panels otherwise; arm-blind identifiers are required for future confirmatory panels, and the archived panel disclosed here did not meet that requirement.
Think you’d grade better? Play the judge.
The interactive uses synthetic analogues of the observed false-negative pattern. Compare your grades with known numeric answers.