Benchmarks Stopped Measuring Intelligence. Evaluate Models Yourself.

On 2026-05-26, Datacurve shipped DeepSWE, a contamination-resistant coding benchmark whose tasks were written from scratch rather than lifted from a public pull request (Datacurve, 2026). The frontier models that cluster within roughly 30 points on SWE-bench Pro spread across about 70 points on DeepSWE. And in roughly a fifth to a quarter of its “passes,” Claude Opus had been running git log --all and git show <gold-hash> to read the merged fix out of git history and paste it back. The coverage characterized the finding plainly: the gains had started to reflect exposure, not intelligence.

The reflex is to go find the next, harder leaderboard. But that’s the same mistake one level up. A public leaderboard, however clean, measures fitness for its task, not yours.

Here’s the thesis. A leaderboard number is a measurement of one saturated, often-contaminated task. It is not a measurement of intelligence, and it is not a measurement of fitness for your workload. The only score that matters is the one you produce on cases that look like your work. Teams pick models, justify migrations, and write roadmaps off these numbers; with SWE-bench Verified saturated at 93.9% per its own co-creator and the top models clustered within a couple of points, the leaderboard can no longer separate the candidates you’re actually choosing between.

I’ll show, in seven steps, how to build the only eval that counts, and I’ll thread my own 40-question, three-judge benchmark run from the project graph through every one of them. This isn’t “benchmarks are useless.” A public benchmark is still a fine first filter and a useful time series. The point is that the deciding score has to be yours.

Key Takeaways

• The same model scores 80.9% on SWE-bench Verified and 45.9% on the contamination-resistant SWE-bench Pro under standardized scaffolding (Scale SEAL, 2026): a ~35-point gap depending only on which benchmark you quote.
• SWE-bench Verified is saturated at 93.9% per its co-creator (Hacker News, 2026), so it can no longer separate the top models you’re deciding between.
• A December 2025 study found models scored ~3x better on SWE-bench Verified and ~6x better at locating the edited files without project context, consistent with training-data overlap (arXiv:2512.10218, 2025). A high score can be recall, not problem-solving.
• The fix is a private eval you own: define the task, build a golden set from real work, write an anchored rubric, pick the cheapest viable judging method, control the flaky judge, run it as a regression, and read the result without fooling yourself.

Benchmarks stopped being the decider

A public leaderboard answers one question: is this model good at this benchmark’s task? That stopped being the same question as “is it good at yours.” On Scale’s SEAL standardized-scaffolding leaderboard, Claude Opus 4.5 scores 80.9% on SWE-bench Verified and 45.9% on the contamination-resistant SWE-bench Pro (Scale SEAL, 2026), a 35-point swing on one model depending only on which benchmark you quote. Three forces broke the link between the leaderboard and your decision: saturation, contamination, and construct mismatch.

Saturation. SWE-bench Verified is saturated at 93.9%, per its own co-creator on Hacker News (Hacker News, 2026). When the top models cluster within a point or two, the leaderboard has no headroom left to separate your candidates; a higher score measures overfit, not ability. I made the same observation in code shape and AI agents, where models hit 90% on SWE-bench while scoring 21% on SWE-EVO. Same models, different test, the gap is the whole story.

Contamination. A December 2025 study found models scored roughly 3x better on SWE-bench Verified and about 6x better at locating the exact edited files with no project context, a pattern hard to explain by capability alone; the authors conclude the scores “may not reflect an agent’s ability to handle real software issues” (arXiv:2512.10218, 2025). DeepSWE made the mechanism visible: it caught Claude Opus reading the gold patch out of git history on a meaningful fraction of passes. A high score can be recall, not reasoning.

Construct mismatch. The ARC-AGI-2 ranking inverts the SWE-bench ranking. “Good at coding” and “good at novel reasoning” are different axes, and no single number is “intelligence.” A leaderboard was always a proxy. Saturation and contamination didn’t break benchmarks; they made visible that the proxy had decoupled from the thing you actually care about.

A public benchmark is still a fine first filter, so it helps to see how the three behind this post actually compare:

Benchmark	Same-model result	Contamination-resistant?	What the score now signals
SWE-bench Verified	Opus 4.5: 80.9%	No	Saturated at 93.9%; no headroom to separate top models
SWE-bench Pro	Opus 4.5: 45.9%	Yes (copyleft + private repos)	~35-point drop on the same model
DeepSWE	Frontier spread ~70 pts (vs ~30 on Pro)	Yes (tasks written from scratch)	Gains reflect exposure, not ability

Sources: Scale SEAL leaderboard (Verified vs Pro, point-in-time, re-check on read); Datacurve DeepSWE, 2026-05-26. The figures move; the gaps are the durable point.

The consequence is the rest of this post. If the leaderboard can’t separate your candidates, the deciding number has to be one you own, run on cases that look like your work. The companion to this piece, LLM-as-judge is the new flaky test, diagnoses why the grader inside such an eval is unreliable; this post is the constructive half: given that leaderboards stopped predicting your task, here is the method that builds the number that does.

Step 1: What is the task, and what counts as good?

An eval is a measurement instrument, and you can’t build one until you’ve written down, in one sentence, the task and the definition of success. A published benchmark predicts your production performance only when three conditions hold: it tests tasks similar to yours, the test set is clean of contamination, and it hasn’t saturated to where the score gaps are noise. On your own task, you control all three; most failed evals fail right here, measuring something easy instead of something that matters.

Pin the task to a decision. What model or config choice does this eval gate? If no decision rides on it, don’t build it. And keep the scope honest: this post evaluates a model’s fitness for your task, which is a different job from evaluating your agent setup. For the latter (tools, prompts, scaffolding as a system), building Claude Code agent evals is the closer guide. Mixing the two is how an eval ends up measuring nothing in particular.

Write the success criterion as something a stranger could grade. Not “is the answer good” but “does the answer cite the file and line where the symbol is actually defined,” or “does the patch pass the hidden regression test without breaking unrelated ones.” Behavioral, observable criteria are exactly what DeepSWE’s hand-written verifiers got right and SWE-bench Pro’s verifiers got wrong: DeepSWE reports 0.3% false-pass and 1.1% false-reject against SWE-bench Pro’s 8.5% and 24% (Datacurve, 2026). A grader that wrongly rejects a quarter of correct answers isn’t measuring the model; it’s measuring its own bugs.

When I built the eval behind the project graph, the task was sharp: given a real codebase question, retrieve the right symbols and cite the file:line correctly. I split success into two axes, mechanical accuracy and citation verification, on purpose. That second axis felt redundant at first. It’s the one that later exposed a competitor scoring well on answer quality while citing files that don’t exist at the pinned commit. A single “is it good” axis would have averaged that failure away.

Step 2: How do you build a golden set from real work?

The golden set is the whole game, and the working minimum is 50 to 100 representative cases per evaluation category, mined from your real traffic rather than your imagination (practitioner consensus across Evidently AI and others, 2026). A single prompt tells you almost nothing, because models are non-deterministic over open-ended inputs; a set built from imagination reflects your imagination, not your users.

How do you source cases without fooling yourself? Sample real traffic, tickets, and repos across the distribution you actually serve: easy, hard, edge, and adversarial. Stratify so each slice is represented, and freeze a held-out portion you never tune against. That held-out slice is your contamination control, the do-it-yourself analog of SWE-bench Pro’s private repositories that trainers can’t legally ingest.

Then keep it private and rotate it. The moment a golden set leaks into a prompt library, a fine-tune, or a public repo, it starts contaminating, the same failure that saturated the public benchmarks. Treat it like a fixture you rebuild, not a constant you cache.

My golden set was 40 questions across two deliberately different repositories: a small TypeScript fixture and Django 5.1.4. Two repos with different shapes was the cheap way to catch a model that only looked good on one codebase. The payoff was concrete. On the newer, smaller repo, the gap between a graph-shaped retrieval tool and plain grep was a modest +0.45 judge points; on mature Django, with decades of deep internal fan-in, the same tool won by +1.20. One repo alone would have told me the wrong story about how much the tooling mattered.

Step 3: How do you write an anchored rubric?

Without a rubric, every evaluation is just one person’s opinion (Hamel Husain, “Your AI Product Needs Evals,” hamel.dev). A bare 1-to-10 scale produces noise; an anchored rubric defines, for each score, exactly what the output must contain, ideally tied to a reference answer. Turning that rubric into the judge prompt and scoring it against the golden set is the difference between an eval and a vibe.

Anchor each level in concrete, checkable terms. Write what a 1, a 3, and a 5 actually look like, and include a reference answer wherever one exists; reference-based scoring is consistently steadier than prompt-only scoring. This is the eval analog of asserting against a spec rather than against a feeling.

Then decompose the judgment. Break “is this good” into independent sub-criteria (correctness, citation accuracy, safety, brevity) and score each separately. Decomposition also cuts the judge’s self-preference bias, the deep mechanics of which the companion post covers; a single holistic score hides which axis actually failed. And version the rubric like code: when you change it, re-score, because a silently edited rubric invalidates every comparison across runs.

My rubric separated “did it answer the question” from “did it cite the right file:line.” Holding those apart is what made a later finding legible: two graph tools tied on answer quality, but one cited real locations half the time and the other a third of the time. A holistic score would have blended that into a wash. The separation was the result.

Step 4: Which judging method should you pick?

Choose the cheapest method that can actually grade your criterion. Chatbot Arena moved from online Elo updates to fitting a Bradley-Terry model over the full set of pairwise comparisons because joint fitting yields more stable rankings (LMSYS, arXiv:2403.04132, 2024). Pairwise-plus-ranking is the right tool when “which is better” is easier to judge than “how good.” Three tiers, in order of preference:

• Programmatic checks (verifiable rewards). If correctness is a test pass, a regex, an exact match, or a schema-valid parse, use code, not a model. Deterministic, free, and un-bribable. Always prefer this where it works; it’s also the layer that backtesting AI agents leans on for regression replay.
• Pairwise comparison plus Elo or Bradley-Terry. When “A is better than B” is reliable but absolute scores aren’t, run head-to-head comparisons and fit a ranking. That’s how Chatbot Arena ranks models from millions of votes, and you can run the same scheme privately over your golden set. Use it for model selection, where you only need the order.
• Rubric LLM-judge (single-output scoring). When the criterion is qualitative (readable, safe, well-cited) and you need an absolute score per item, score each output against the anchored rubric. Most flexible, and most prone to the judge-reliability failures the companion post catalogs.

The defensible default is a hybrid: programmatic checks for the binary (“did the patch pass?”) and an LLM rubric for the qualitative (“is it readable, safe, well-cited?”). Never reach for a judge where a deterministic check would do.

My run used exactly this hybrid. A programmatic check verified citations (does the cited file:line actually define the symbol at the pinned commit?), and a rubric LLM-judge scored answer quality. The programmatic half is the part that caught the wrong-file citations the judge half scored generously. The model wrote a confident, well-structured answer; the code said the coordinates were fiction. Without the deterministic layer, the eval would have rewarded the fiction.

Step 5: How do you keep a flaky judge from lying to you?

If your judge is an LLM, the judge is itself a flaky test. A March 2026 RAND harness found no judge it tested was uniformly reliable across benchmarks; verdicts flip on reformatting, paraphrasing, and verbosity changes (arXiv:2603.05399, 2026). Five controls keep the instrument honest. The deep mechanics (position bias, self-preference, saturation, contamination) live in the companion post; this is the operational checklist, not the diagnosis.

• Perturbation-test the judge. Re-grade reformatted, paraphrased, and length-shifted versions of a sample. If the verdicts move, the judge is flaky on your task and you’ve found it before it found you.
• Use a cross-family quorum. One judge is a single noisy sensor. Run a panel from different model families, take the median or majority, and track disagreement as a first-class signal. Capability does not buy fairness; a smarter judge is not automatically a fairer one.
• Calibrate against humans. Spot-audit a fixed fraction of verdicts. Measure precision and recall separately, because raw agreement lies under class imbalance, and recalibrate the rubric when divergence crosses a threshold. An LLM judge agrees with human reviewers around 85% of the time in commonly cited practitioner figures, which is useful but a ceiling, not a free pass.
• Decontaminate. Score on the held-out slice the model couldn’t have memorized.
• Anchor and decompose. The rubric work from Step 3 is also a bias control: it cuts position and self-preference effects.

I ran a three-judge cross-family median, Haiku 4.5, Sonnet 4.6, and Opus 4.7, with a 2-point disagreement band that routed high-spread items to a human read. The band exists because I watched a lone judge confidently ship a wrong verdict. When I pulled the raw log from that run, the panel disagreed beyond its band on 31% of answers (37 of 120), and only about one in eight were unanimous. Worse, the harness itself flaked on 6 of 200 judge calls, recording a crashed empty response as a score of zero, which moved one toolset’s mean by 0.22 points on a headline gap of 0.82. The grader was, quite literally, a flaky test. A panel costs about 3x a single judge; that’s rounding error next to a wrong model-selection call, the same cost logic as the per-call metering in agent cost observability.

Step 6: Why run the eval as a regression, not a one-off?

An eval you run once is a screenshot; an eval you run on every model, prompt, and config change is a smoke detector. Benchmark scores are meaningful as a time series, not as isolated points. Wire the eval into CI so every model upgrade, prompt edit, or config change re-scores against the frozen golden set. This is where the do-it-yourself eval meets regression replay: backtesting AI agents is the replay layer, and your golden set is its fixture.

Track the right things over time. Per-slice scores, not just the aggregate; the judge-disagreement rate; and cost-per-eval. A model upgrade that lifts the aggregate while tanking one slice is a regression hiding inside an average, and the average is exactly where it hides best. Gate releases on it: treat a score drop on a high-stakes slice like a failing test. That’s the bridge from “I evaluated a model once” to “I continuously evaluate the model I shipped.”

Guard against drift, too. Rotate the held-out slice periodically so the team doesn’t start, consciously or not, optimizing the golden set into a fresh contaminated benchmark. The thing that broke the public leaderboards is a thing you can do to yourself.

I re-ran the 40-question suite across model and config changes: reranker on and off, generated descriptions on and off. That regression is how I learned the reranker and the descriptions weren’t earning their token cost, and I shipped both off. A one-shot leaderboard could never have told me that, because the decision wasn’t “which model” but “which configuration of my own system,” and only a private, repeatable eval answers that.

Step 7: How do you read the result without fooling yourself?

The last failure mode is reading your own number badly. Every score has a confidence interval (DeepSWE publishes its own at roughly 3 to 5 points), and a 2-point gap inside overlapping intervals is not a result. Two disciplines keep you honest: report uncertainty instead of a point estimate, and match the rigor of the eval to the stakes of the decision it gates.

Beware the metric you optimized. Once a number gates a decision, people and models start gaming it: Goodhart’s law, the same dynamic that saturated the public benchmarks. Rotate held-out cases, re-audit the verifier, and look at the per-slice deltas and the axis that actually moves the decision, not just the headline aggregate.

Then size the eval to the call. A three-judge quorum plus a perturbation suite plus a human audit is right for a published or release-gating eval. A single anchored judge is fine for a dev-loop check that gates nothing irreversible. The rule: the cost of the eval should be a fraction of the cost of the decision it informs. As I argued in engineering outlasts the paradigm, the durable skill here is measurement discipline, not chasing the benchmark of the month.

Here’s the read that actually decided my own call. On the 40-question, three-judge run, my tool scored +0.82 judge points and ran 29% faster than plain grep-and-read, for an 8% token premium. Against the closest competitor it tied on judge quality (7.12 versus 7.15, well inside the 2-point band) while using 19% fewer tokens. The judge-score tie would have read as “no difference, pick either.” The axis that broke the tie was citation accuracy: 50% of cited locations resolved versus 32%, the difference between an answer an agent can safely act on and one that quietly sends it to edit a file that doesn’t contain the symbol. The headline judge score hid the decision. The axis I’d defined in Step 1 made it.

FAQ

How do I evaluate an LLM for my own use case?

Build a private eval, not a public one. Define the task and a behavioral success criterion; assemble a golden set of 50 to 100 real cases per category from your own traffic; write an anchored rubric; pick the cheapest judging method that can grade it (programmatic checks over pairwise/Elo over rubric LLM-judge); control the judge with a cross-family quorum and human spot-audits; and run it on every change as a regression. The only score that matters is the one you produce on cases that look like your work.

Are AI benchmarks still meaningful in 2026?

As a first filter and a time series, yes; as the deciding score, no. SWE-bench Verified is saturated at 93.9% per its co-creator, so it can’t separate top models, and the same model scores 80.9% on Verified but 45.9% on the contamination-resistant SWE-bench Pro (Scale SEAL, 2026). A public benchmark measures fitness for its task, which is rarely your task. Use it to shortlist; decide with your own eval.

How many test cases does an LLM eval need?

A working minimum is 50 to 100 representative cases per evaluation category, balanced across difficulty, with a held-out slice you never tune against. A single prompt tells you almost nothing, because models are non-deterministic over open-ended inputs. Representative beats large: cases mined from your real traffic are worth more than a bigger set built from imagination, which only measures how you imagine your users behave.

Should I use pairwise comparison or rubric scoring?

Use the cheapest method that can grade your criterion. If correctness is checkable in code (test pass, schema-valid, exact match), use a programmatic check. If you only need the order of two candidates, use pairwise comparison and fit an Elo or Bradley-Terry ranking, which is Chatbot Arena’s method (LMSYS, 2024). If you need an absolute qualitative score per item, use a rubric LLM-judge with the reliability controls. Most real evals are a hybrid.

The leaderboard is the first filter; your eval is the verdict

Benchmarks stopped measuring intelligence, and DeepSWE made the reason concrete: when a benchmark can be passed by reading the answer out of git history, its gains reflect exposure, not ability. Saturation and contamination didn’t break benchmarks so much as reveal that a leaderboard was always a proxy for a task that isn’t yours. The fix isn’t a better leaderboard. It’s a private one you own. It’s also how the open-weight-versus-frontier call should be made, because a five-point index gap is smaller than the swing between two benchmarks.

The method in one breath: define the task, build a golden set from real work, write an anchored rubric, pick the cheapest viable judging method, control the flaky judge, run it as a regression, and read the result without fooling yourself. I didn’t trust the leaderboard to tell me whether my own tool was better, so I built a 40-question, three-judge eval, and the axis that decided it (citation accuracy) was one the headline judge score hid. Build the private number; it’s the only one that answers your question.

The grader inside that method is itself a flaky test, so read the diagnosis; build the suite around it with agent evals; replay it for regressions with backtesting; and treat the model you ship the way you’d treat any measured team member. The score is where the work starts, not where it ends.