LLM-as-Judge Is the New Flaky Test
On 2026-06-08, the co-creator of SWE-bench posted on Hacker News: “SWE-bench Verified is now saturated at 93.9% (congrats Anthropic)” (Hacker News, 2026). The benchmark that defined coding-agent progress, climbing from roughly 4% to 90%-plus in under three years, is done as a discriminator. Its own author is shipping successors.
The reflex is to go find a harder benchmark. That reflex misses the deeper problem. When the answer isn’t a passing unit test, somebody has to score quality, and increasingly that somebody is another language model. The grader is an LLM, and the grader is unreliable.
That’s the thesis: LLM-as-judge is the new flaky test. A green eval can be measuring judge bias, position, verbosity, self-preference, or training-data memory rather than the ability you think you’re scoring. A March 2026 RAND harness found that no judge it tested was uniformly reliable (arXiv:2603.05399, 2026). Meanwhile teams are promoting models, gating releases, and writing roadmaps off these numbers. If the grader flips on a paraphrase and the benchmark has no headroom left, the decision is noise wearing a green hat.
I’ll walk three failure surfaces, the judge, the benchmark, and the data, then give the five-part discipline that makes LLM-as-judge trustworthy. Along the way I’ll audit my own three-judge method from the-project-graph, including the parts where it caught itself being wrong. This isn’t “go back to humans.” Used with discipline, LLM-as-judge is the best scalable evaluation we have. The point is the discipline, not the abandonment.
Key Takeaways
- A March 2026 RAND “Judge Reliability Harness” reported that “no judge that we evaluated is uniformly reliable across benchmarks”; verdicts flipped on formatting, paraphrasing, and verbosity changes (arXiv:2603.05399, 2026).
- Capability doesn’t buy fairness: across 20 models, advanced ability was uncorrelated, sometimes negatively correlated, with low self-preference bias (arXiv:2604.22891, 2026).
- SWE-bench Verified saturated at 93.9%, yet the same top model scores ~45.9% on the contamination-resistant SWE-bench Pro: a ~35-point gap on one model, depending only on which benchmark you quote.
- In my own three-judge run, the panel disagreed by more than its 2-point band on 31% of answers (37 of 120), and the harness itself flaked on 6 of 200 judge calls. The fix is a discipline, not a vendor.
Is LLM-as-a-judge reliable? Not by default
A single LLM judge behaves like a flaky test: the same response can be scored pass or fail depending on formatting, phrasing, and length that have nothing to do with quality. A March 2026 RAND harness put four state-of-the-art judges against four benchmarks and concluded, verbatim, “No judge that we evaluated is uniformly reliable across benchmarks” (arXiv:2603.05399, 2026). Consistency broke on simple text-formatting changes, paraphrasing, shifts in verbosity, and flipping the ground-truth label.
Sit with what that means. You hand a judge an answer and get a 7. You reformat the same answer into bullet points and get a 5. You pad it with a polite preamble and get an 8. Nothing about the underlying quality moved. The sensor did.
Why does this happen? Position bias is one mechanism. “Judging the Judges” measured it across 15 judges, roughly 40 models, 22 tasks, and 150,000-plus evaluations, and found position bias is only weakly tied to answer length but strongly driven by the quality gap between the two solutions being compared (arXiv:2406.07791, 2024, rev 2025). When two answers are close in quality, the judge’s preference drifts toward whichever it saw first. The closer the call, the less the judge is judging.
In continuous integration, a test that flips on a re-run with no code change gets quarantined, not trusted. An LLM judge that flips on a paraphrase is the same artifact. We just haven’t built the quarantine discipline yet.
What is self-preference bias in LLM judges?
Using a frontier model as your judge does not buy you a fair judge. Across 20 mainstream LLMs, advanced capability was uncorrelated, and sometimes negatively correlated, with low self-preference bias, the tendency to favor outputs that resemble the judge’s own (arXiv:2604.22891, 2026). Smarter graders are not automatically fairer graders, and some are worse.
The common shortcut is “use the smartest model as the grader.” Walk through what that does to a head-to-head model comparison. If your grader systematically prefers responses that share its family’s style, you aren’t measuring quality. You’re measuring stylistic kinship, and you’re handing an advantage to whichever contestant writes most like the referee. Self-preference is the eval analog of a test that passes because it asserts against its own buggy fixture. Green, and circular.
The same paper offers a concrete lever, not just a warning. Decomposing the judgment, breaking one holistic score into smaller structured sub-judgments under lower cognitive load, cut self-preference bias by 31.5% on average (arXiv:2604.22891, 2026). That’s a real mitigation: stop asking for a vibe-based 1-to-10 and start asking targeted questions whose answers compose into a score.
There’s a cheaper guard that follows directly: don’t let a model grade its own family in a comparison that matters. If you’re scoring Claude outputs, put a non-Claude model on the panel, or at least cross-check a same-family judge’s verdicts against a different-family one. When the two disagree systematically in the home model’s favor, you’ve found your bias, and the size of that gap is roughly the correction you owe the loser. It’s also why a single-judge leaderboard a lab runs on its own model deserves a raised eyebrow. The referee and one of the players share a training lineage.
This bias compounds the failure surfaces I catalogued in autonomous-agent-failure-modes. A judge that quietly prefers its own family is a detection-evasion mode: the eval looks healthy precisely because the grader is rewarding the wrong thing.
Is SWE-bench saturated? Yes, and the green number is stale
Even a perfectly reliable judge can’t rescue a saturated benchmark. When a test has no headroom left, a higher score measures overfit, not ability. SWE-bench Verified climbed from roughly 4% to 90%-plus in under three years, and on 2026-06-08 its co-creator called it “saturated at 93.9%” and pointed to successors like CodeClash and AlgoTune (Hacker News, 2026). When the benchmark’s own author declares the finish line crossed, the number stops discriminating.
Here’s the story in one comparison. On Scale’s SEAL standardized-scaffolding leaderboard, Claude Opus 4.5 scores 80.9% on SWE-bench Verified but only 45.9% on SWE-bench Pro (Scale SEAL, pulled 2026-06-27). SWE-bench Pro is contamination-resistant by design: 1,865 longer-horizon tasks, averaging ~107 lines changed across ~4.1 files, drawn from copyleft and private repositories that trainers can’t legally ingest (arXiv:2509.16941). A ~35-point gap on the same model is the whole argument. The number you quote depends entirely on which benchmark you pick, and the harder one isn’t saturated.
| Benchmark (same model: Claude Opus 4.5) | Score | Contamination-resistant? | Signal |
|---|---|---|---|
| SWE-bench Verified | 80.9% | No | Saturated (93.9% top score) |
| SWE-bench Pro | 45.9% | Yes (copyleft + private repos) | ~35 points of headroom remain |
Source: Scale SEAL standardized-scaffolding leaderboard, pulled 2026-06-27. Re-pull before quoting.
These numbers move, so treat them as point-in-time. As of 2026-06-08, the top public SWE-bench Pro result was ~77.8% (Claude Mythos Preview), and Terminal-Bench v2 (89 tasks) had Opus 4.7 around 69.4%. Pull them again before you quote them; the gap is the durable point, not the decimals. For real-money grounding, SWE-Lancer assembled 1,400-plus freelance tasks worth $1M and found frontier models solved only a minority of them (arXiv:2502.12115, 2025).
A saturated benchmark is a test that passes for everyone, so it no longer tells anyone apart. In CI you’d delete or rewrite it. In evals we keep quoting it on slides. The headroom gap here echoes code-shape-and-ai-agents (90% on SWE-Bench against 21% on SWE-EVO) and the pass@1 collapse over long horizons in long-running-autonomous-agents.
Is a green eval measuring ability or memory?
A green eval can be scoring recall, not reasoning. If the test cases were in the training data, the model recites the patch instead of deriving it. A December 2025 study found Claude models scored roughly 3x better on SWE-bench Verified and about 6x better at locating the exact edited files without any project context, a pattern hard to explain by capability alone (arXiv:2512.10218, 2025). The authors conclude the scores “may not reflect an agent’s ability to handle real software issues.”
Locating the precise file to edit, with no repository context provided, is the tell. That’s not problem-solving; that’s memory of where this specific bug lives. When a model can point straight at django/contrib/auth/__init__.py for a known issue it was never shown in-session, you’re watching retrieval of a memorized fixture, not analysis.
The structural fix mirrors decontaminating a test fixture so it can’t pass by caching the answer. Hold out tasks the model couldn’t have seen, time-box them to after the training cut, and prefer benchmarks built on private or copyleft repositories that trainers can’t legally ingest. That last property is exactly SWE-bench Pro’s design.
You can run a cheap contamination probe yourself, borrowing the paper’s own tell. Strip a task down to its issue text, give the model no repository access, and ask which files it would edit. A model reasoning from the description lands in the right neighborhood sometimes. A model that names the exact changed files, repeatedly, with no way to have inferred them, is reciting rather than solving, and any score built on those tasks is measuring memory. One widely-shared claim goes further: that 59.4% of audited SWE-bench problems have flawed test cases that reject functionally correct submissions. That figure comes from a Hacker News commenter, not the benchmark’s authors, so treat it as disputed and unverified until someone sources it.
Contamination is the test that passes because the answer leaked into the fixture. Decontamination is rebuilding the fixture from scratch so a pass has to be earned.
How do you make an LLM judge trustworthy? Port the flaky-test playbook
Engineers already solved “untrustworthy green” once. Flaky tests forced a discipline, quarantine, vote, triage, fix the fixture, assert against a spec, and that discipline ports directly onto LLM-as-judge. Single-judge evaluations are too unstable to trust on their own; the practitioner consensus through 2026, backed by the position-bias and self-preference findings above, is that panels and structure beat a lone holistic score.
Each mapping is operational, not metaphorical:
- Quarantine the flake becomes perturbation-test the judge. Before you trust a judge, fuzz it: re-grade the same response reformatted, paraphrased, and length-shifted. If the verdict moves, the judge is flaky on this task and the RAND harness is the off-the-shelf tool to prove it.
- Re-run N times and vote becomes a multi-judge quorum. One judge is a single noisy sensor. Run a panel, take the median or majority, and track disagreement as a first-class signal.
- Manual triage becomes a human spot-audit. Sample a fixed fraction of verdicts for human review, track divergence, and recalibrate the rubric when divergence crosses a threshold (~20-25% is a commonly cited trigger).
- Fix the fixture becomes decontaminate and hold out. Use tasks the model couldn’t have memorized, rotate held-out sets, and prefer contamination-resistant benchmarks.
- Assert against a spec becomes an anchored rubric. Replace “rate this 1-to-10” with explicit, anchored criteria and, where one exists, a reference answer. Reference-based scoring is consistently steadier than prompt-only vibes.
One caveat keeps this honest: where a deterministic check works, use it. Verifiable rewards (does the code pass the test, do the cited file:line locations actually exist?) answer the binary cleanly. LLM rubrics answer the qualitative (is it readable, safe, well-structured?). The defensible default is the hybrid, and you never reach for a judge where a unit test would do. This is the operational layer beneath claude-code-agent-evals, which builds the suites, and backtesting-ai-agents, where replay scoring leans on LLM-as-judge and therefore inherits every risk in this post.
My own method, audited: three judges, median, a 2-point band
I don’t trust a single green judge, including my own, and here’s the run that taught me why. In the-project-graph I scored a 40-question benchmark across two repositories with three judges from different families, Haiku 4.5, Sonnet 4.6, and Opus 4.7, taking the median of three and keeping a 2-point disagreement band. The band’s job: when the three judges spread more than 2 points on the 0-to-10 scale, the item isn’t a clean auto-score, it’s flagged for a human read. That’s the quarantine step, made concrete.
When I pulled the raw judge log from that run (R005, 2026-05-28) and counted, the panel disagreed by more than its 2-point band on 31% of answers: 37 of the 120 datapoints across the three compared toolsets. Only 16 of 120, about one in eight, were unanimous. So on roughly a third of items, a single randomly-chosen judge could have landed 3 or more points off the panel’s median, on a 10-point scale. That isn’t a rounding error. That’s a coin you don’t want to flip when the result gates a published decision.
The band earns its keep on individual items. Take django-symbol-03, a question about Django’s get_user_model. The agent’s answer correctly cited django/contrib/auth/__init__.py:183 but stopped there, no explanation of what the function calls or what it raises. Opus scored it 9, reading the rubric leniently: the citation was the mandatory requirement, and the behavioral detail was a bonus. Haiku and Sonnet both scored it 4, reading the same rubric strictly: a code-intelligence answer that gives coordinates without behavior is, in Sonnet’s words, “severely incomplete.” The median came out 4. A lone Opus judge would have shipped a 9 for an answer two other judges called barely passing, and the disagreement wasn’t noise; it was a genuine split over how strictly the rubric binds. The quorum caught what a single confident judge would have waved through. The outlier runs both directions, too: on one architecture answer, Haiku was the harsh dissenter at 3 while Sonnet and Opus scored 8 and 9, and the median again pulled the score back to honest.
Then there’s the part nobody puts in a methods section: the grader itself flaked. On 6 of 200 judge calls in that run, one judge returned an empty justification scored as a 0, a silent crash recorded as a verdict of zero. If you average those raw zeros in, the default toolset’s judge mean reads 6.08. Drop the crashes (which is why the published chart in that post is labelled “median, clean”) and it’s 6.30. A 0.22-point swing, on a benchmark whose headline gap was 0.82, sitting entirely on whether you let a flaky harness vote. The grader was, quite literally, a flaky test. I’d written the quorum to defend against biased judges and discovered it also had to defend against crashed ones.
Why pay for three judges? A cross-family panel partially cancels family-specific biases, self-preference and style preference don’t all point the same way across Haiku, Sonnet, and Opus, and the median is robust to any one judge being an outlier on a given item. The honest limit: median-of-three is not a guarantee. Three judges that share a failure mode (all rewarding verbose answers, say) will agree and be wrong together, which is exactly why the human spot-audit is non-negotiable rather than optional. The cost is real, three judges run about 3x a single judge, but next to a wrong model-selection call it’s rounding error, the same logic as the per-call metering in agent-cost-observability-team.
When is a single judge fine?
Not every eval needs a quorum, a perturbation suite, and a human audit; match the rigor to the stakes. A single anchored judge is fine when the score gates nothing irreversible, when a verifiable check could back it up, when you re-run often enough that one wrong verdict self-corrects, and when the rubric is anchored and the answer space is narrow. Over-building the grader for a dev-loop check is its own kind of waste.
The full discipline earns its cost when the result is published, gates a release, drives model selection or spend, or feeds a leaderboard other people will trust. The rule is simple: the cost of the discipline should be a fraction of the cost of the decision the eval informs. A 3x judge bill to avoid a wrong six-figure model-migration is trivial. The same bill to pick between two prompt variants in a scratch loop is theater.
That framing is the durable skill, not the benchmark of the month. As I argued in engineering-outlasts-paradigm, the half-life of any specific score is short; the half-life of measurement discipline is your whole career.
FAQ
Is LLM-as-a-judge reliable?
Not by default. A March 2026 RAND harness found no judge it tested was uniformly reliable across benchmarks; verdicts flipped on formatting, paraphrasing, and verbosity changes (arXiv:2603.05399, 2026). It becomes reliable only with discipline: anchored rubrics, a multi-judge quorum, perturbation tests, human spot-audits, and decontaminated tasks. Treat a single green verdict the way you’d treat a single green run of a flaky test.
What is self-preference bias in LLM judges?
It’s a judge systematically favoring responses that resemble its own outputs. Crucially, it doesn’t shrink with capability: across 20 mainstream models, advanced ability was uncorrelated, sometimes negatively correlated, with low self-preference bias (arXiv:2604.22891, 2026). A cognitive-load-decomposition grading strategy cut it by 31.5% on average. Using the smartest model as judge does not buy you a fair judge.
Is SWE-bench saturated?
Yes, for SWE-bench Verified. On 2026-06-08 its co-creator called it “saturated at 93.9%” and pointed to successors (Hacker News, 2026). The tell: the same top model scores ~80.9% on Verified but ~45.9% on the contamination-resistant SWE-bench Pro. The benchmark no longer discriminates, so a higher score measures overfit, not ability. Re-check leaderboard numbers on read.
How many LLM judges should you use?
For anything that gates a release, drives model selection, or gets published, use a panel of three from different model families and take the median, with a disagreement band that routes high-spread items to human review. I used Haiku 4.5, Sonnet 4.6, and Opus 4.7 with a 2-point band; the panel disagreed beyond it on 31% of answers. A panel costs about 3x a single judge, cheap next to a wrong decision. For low-stakes dev-loop checks, one anchored judge is fine.
The green checkmark is a hypothesis, not a result
LLM-as-judge is the new flaky test, and a saturated or contaminated benchmark is the test that no longer tells anyone apart. Three failure surfaces, one line each: the judge flips on trivial perturbations, the benchmark has no headroom, and the data may be memorized. None of them announce themselves. All of them render green.
The way out isn’t a vendor; it’s a discipline you already half-know from CI. Anchored rubrics, a multi-judge quorum, perturbation tests on the grader, human spot-audits, and decontaminated held-out tasks. I run three judges with a disagreement band and still spot-audit, because I’ve watched a lone judge confidently ship a 9 the panel scored a 4, and I’ve watched the judge harness itself crash and record the crash as a verdict. Distrust your green number until you’ve stress-tested the thing that produced it.
So build the suite (claude-code-agent-evals), replay for regressions (backtesting-ai-agents), and treat the judge itself as a measured team member (treat-ai-as-a-team-member), not an oracle. The score is where the work starts, not where it ends.
If it was useful, pass it along.
