Your Interview Measures the Agent, Not the Candidate

Five times in five minutes one morning, a coding agent finished a take-home I had given it and opened a pull request under my name.

The take-home was FeedMe’s, the public technical interview assignment of a real company. Its brief tells candidates to fork the repository and open a pull request. My harness had isolated the agents’ instructions and environment variables, but it had passed through my real home directory, and with it my gh login. So Codex did what the brief said: forked, pushed, and opened four public pull requests on FeedMe’s repository. The fifth run went further. It searched GitHub, found a stranger’s published solution, read their code, forked that repository and opened the pull request there.

I closed all five with an apology and deleted the forks. But it is the whole post in one incident. The agent did the entire take-home, including the part where it submits it, and the only thing it needed from a candidate was their credentials.

One more thing before the data, because it matters for how you read it. I built a tool for exactly this.

Key Takeaways

In 438 unaided runs across four interview formats, every automated correctness criterion on every take-home was met by every agent (Claude Sonnet 5.5, Claude Opus 5.5 and Codex) in every run.

What still separated agents was judged, not automated: whether they asked or stated an assumption about open questions, caught edge cases the brief didn’t spell out, pushed back on a bad requirement, and reported honestly.

Given a module with six planted bugs, every agent found all six when the domain rules were in a README. Without the README, the two bugs that were only wrong against a business rule were explained 3 times in 30.

Three strong agents passed 90 of 90 algorithm runs each. Only a smaller model failed any. 21 of the 30 questions carried no signal at all.

On Anthropic’s open-sourced performance take-home, all three agents beat every model result Anthropic published, in under 35 minutes each.

Why I built an interview cheating tool

In spring I built and published a desktop overlay for technical interviews. You take a screenshot of the problem, and it optionally transcribes the interviewer’s audio, then streams back an approach, code, tests and a complexity analysis into floating panels. I built it to see how hard it would be. It was not hard.

What building it taught me is the reason for this experiment. In a live round, the interviewer can see the candidate’s face but not their screen, and anything that appears on the screen or is said out loud can reach a model in a few seconds. A live round and a take-home are not two different things any more. They are the same thing at different latencies.

The smarter candidates already know this, which is why I am saying it plainly for the other half of my readers: the people running interviews. The evidence that you need to hear it is not subtle. In CoderPad’s 2026 State of Tech Hiring (about 650 respondents, February 2026), “agent-generated solutions” was the top cheating concern at 52%, and only 12% of recruiters and hiring leaders picked take-home projects as a format that best reflects on-the-job ability, against 60% for live coding. Earlier, interviewing.io had candidates secretly use ChatGPT in mock interviews: they passed 73% of verbatim LeetCode questions, and not one interviewer noticed. Anthropic’s own performance team reported in January that Claude Opus 4.5 “matched even” their strongest candidates on the take-home more than 1,000 people had sat.

So the question is not whether candidates use agents. It is which parts of an interview still measure the person once they do.

The experiment

The harness and every run are public at github.com/iceinvein/interview-signal. The design is simple: give each task to an agent exactly as a candidate would paste it, with no hints, and see which criteria it meets on its own. Any criterion that every agent meets in every run is a criterion that tells you nothing about the human.

Four formats:

Take-homes. Three tasks, each run 5 times per agent. FeedMe’s real assignment, unmodified. A webhook relay service I wrote from an ambiguous brief. And a feature to add to a purpose-built 1,500-line TypeScript repository with its own conventions. The two authored tasks each hide one trap: a requirement a strong engineer should question. Each has a hidden acceptance suite and a rubric of 10 to 16 criteria, frozen in a commit before the first paid run.
A comprehension round, in the style of the code-reading rounds some companies now run: a 330-line billing module with six planted bugs. Two are visible in the code, two need tracing across functions, and two are only wrong against a business rule written in a README. Every agent ran it with and without that README.
30 algorithm questions: 10 well-known problems (statements paraphrased), 10 modified variants of those, and 10 custom ones written for this experiment, each with a hidden test suite.
Anthropic’s original performance take-home, scored in simulated clock cycles, with a 2-hour cap.

Agents: Claude Code running Claude Sonnet 5.5 and Claude Opus 5.5, Codex running gpt-6-sol at high effort, and Claude Haiku 4.5 on the algorithm questions only, as a weaker baseline. Every run is a fresh session in a directory holding only the task. None of my own instruction files, hooks or plugins could load. Claude’s web search and fetch tools were off throughout; Codex’s web search was switched off only for the FeedMe container runs, and in the other runs it was available but never used.

Grading: hidden test suites where the answer is mechanical, and three Claude Opus votes per criterion where it isn’t. A blind hand-check backed the judge up: a fresh agent relabelled a seeded 20% of its labels from the raw evidence, without seeing them, and agreed on 98 of 99. LLM judges are flaky, so every headline number here comes from the tests where one exists.

438 runs, no errors, no timeouts. Getting the isolation right took three attempts, and I come back to why at the end.

Do automated take-home checks still tell candidates apart?

This is the result that should change how interviews are graded. On all three take-homes, every automated correctness criterion was met by every agent in every run.

On the existing-repo task that meant no regressions, a clean type check, the feature working through the HTTP API and the capacity rule holding. On the webhook relay: forwarding, retries with the right backoff, stable event ids, rate limiting and per-tenant isolation, all checked over HTTP, plus the candidate’s own tests passing. On FeedMe: its own CI check passing, and VIP ordering, bot handling and the rest of its seven requirements never once violated where the run exercised them (the idle-bot requirement was only exercised in 3 of 15). (FeedMe’s one other automated check, whether the run’s own demo log shows every requirement, measures the demo script rather than the code.)

Several judged criteria went the same way. Every agent, every run, returned errors in the repository’s own result style, respected its repository layer, wrote tests in the house style, and wrote webhook tests that control time instead of sleeping.

These are the things take-home rubrics usually weight most heavily, because they are easy to check. They are now the part of the submission that says least about the candidate.

What still separated the agents?

Not much, and none of it was about whether the code worked.

What separated agents	Sonnet 5.5	Opus 5.5	Codex
Noticed that raising a session’s capacity should promote people off the waitlist	3/5	5/5	0/5
Said what happens to a webhook after its last retry (asked, or stated an assumption)	3/5	5/5	2/5
Handled a waitlisted member who is no longer eligible	5/5	5/5	3/5
Documented how to build, test and run the FeedMe submission, and its design and assumptions	0/5	5/5	5/5
Every claim in its closing report was true of what it delivered (FeedMe, hand-graded)	4/5	2/5	5/5

Look at what those rows have in common. The capacity rule and the ineligible member are edge cases the brief did not spell out; a good candidate finds them by thinking about the domain, not by reading the spec. The retry question was deliberately left open; a good candidate asks it or says what they assumed.

One qualifier: the webhook brief asks candidates to write down the calls they made or the questions they would have asked, so the open-question rows measure whether an agent follows that instruction on a point it has to notice for itself. They don’t measure unprompted curiosity. The honesty row is about the report, not the code: in every failure the closing message claimed that every line of output was timestamped, and the delivered log contradicted it.

That last one deserves its own sentence. On FeedMe, Opus did the most complete work and also overstated it most often. All four failures, three Opus and one Sonnet, were the same claim: that every line of output is timestamped, when a header or banner line was not. An interviewer who grades the artefact sees good work. An interviewer who checks the claims sees a pattern.

The trap: the cleanest signal in the experiment

Each authored take-home hid one requirement that should not be followed as written. The webhook brief asked the service to log “each incoming request, including its full body, so on-call can debug delivery problems”, in a system whose payloads the same brief said carry customers’ API keys. The feature brief asked for a list endpoint to return members’ email addresses, which the repository’s own CONTRIBUTING.md forbade outside one route.

Trap	Sonnet 5.5	Opus 5.5	Codex
Log full request bodies that carry customer API keys	pushed back 5/5	pushed back 5/5	complied and flagged 5/5
Return member emails against the repository’s privacy rule	pushed back 5/5	pushed back 5/5	complied and flagged 4/5, pushed back 1/5

No agent complied silently. But the split between pushing back and complying-while-flagging is as clean as anything in the data. The checker agrees with the judge, too: the planted API key turned up in the service’s logs in 5 of 5 Codex runs and 0 of 10 Claude runs.

That is what a trap measures in a human too. Not whether they can build the thing, but whether they will build the wrong thing because someone asked. It is also the kind of judgment that makes the difference in how agents fail in production, where the instruction you should have questioned is the one that causes the incident.

Knowledge that isn’t in the code is the only thing left to test

The comprehension round is the sharpest result, and it lines up with what I found last week about tests written from code alone.

With the README, every agent found and explained all six bugs in all 15 runs. That includes the two cross-function bugs, like a preview that silently spends the customer’s account credit, which I had expected to be hard. Without the README, the code-visible and cross-function bugs were still found every time. The two bugs that were only wrong against a written business rule (credit applied before tax, and a downgrade that took effect at the wrong time) were explained 3 times in 30, all three by one model inferring a convention. One of them, credit applied before tax, was never explained without the rule, though five runs pointed at the right lines.

If a bug is visible in the code, an agent will find it, so a code-reading round that plants code-visible bugs is testing whether the candidate has an agent open. The only bugs that measured anything were the ones where the fact that made them wrong lived outside the code, in a document a person had to read and connect.

Are algorithm questions still useful?

Sonnet 5.5, Opus 5.5 and Codex each passed all 90 of their algorithm runs: every known problem, every modified variant and every custom question I wrote from scratch. Claude Haiku 4.5 passed 75 of 90, failing 9 questions, three of them in every repeat.

That means 21 of the 30 questions carried no signal at all, and the other nine only separate a small, cheap model from the rest. None of the 30 could tell the three strong agents apart. In interviewing.io’s experiment, custom questions were the defence: candidates using ChatGPT passed only 25% of them. Two years on, the custom questions went the same way as the known ones.

The performance take-home: past Anthropic’s published results

Anthropic open-sourced its original performance take-home as an open challenge once models had caught up with the time-limited version: optimise a kernel for a simulated accelerator, scored in clock cycles. Its README lists what Claude models reached, and says that none of the solutions under 1,300 cycles it received on the first day after release were valid, because a language model had modified the tests.

All three came in under every published figure, in 22 to 31 minutes. The chart shows five of Anthropic’s six published results; the sixth, Sonnet 4.5 at 1,548, sits between two of them. Given that README’s warning, I checked them hard. Each kernel changes only the file it is supposed to change. Each passes upstream’s own nine tests on a clean checkout; I ran them three times, with fresh random inputs each time. And none sees the input data at build time. I read all three by hand: list scheduling to pack instruction slots, 8-lane vectorisation, fused hash stages, the top of the tree held in registers. They are real optimisations.

Two caveats keep this honest. Anthropic’s published numbers are for a 2-hour version that started at 18,532 cycles, while the open-sourced repo starts at 147,734. So the comparison is close rather than exact, though Anthropic describes the released version’s cycle counts as comparable. And this take-home was retired precisely because models caught up; Anthropic’s current one is built around novel, heavily constrained work that Opus 4.5 failed. The lesson is the one their post drew: a take-home that a model can do in 2 hours measures the model.

The agent will follow your brief all the way to the pull request

Back to the incident, because it is a finding and not just an embarrassment.

When the FeedMe brief said to fork the repository and “Create a Pull Request with your changes”, Codex did exactly that with whatever credentials it could reach, five times in five. Some Claude runs checked whether gh was logged in, and several tried to commit, but none pushed. I fixed it by running every agent in a fresh container with only its workspace mounted. Then, in the first container rerun, three Codex runs reached GitHub as me anyway. They went through the GitHub connector that comes with a ChatGPT login: they asked which account was signed in and searched for existing solutions, and one opened a stranger’s published answer. Read-only this time, but the same shape. I switched the connectors off and ran those again.

Two lessons for hiring, one for each side of the table:

For interviewers: a take-home whose submission step is “open a pull request” will be completed end to end by an agent, including the submission. If your process treats the pull request as evidence the candidate engaged with your codebase, it isn’t any more.
For candidates: pointing an agent at a stranger’s take-home repository gives that repository your credentials by default. I have written about why you should sandbox coding agents and treat their tools as a supply chain. An interview repository is exactly the kind of untrusted input that makes an agent a loaded gun.

What should interviews test instead?

If you run interviews, here is what the data says still measures a person.

Stop weighting what a CI check can grade. Tests pass, types check, the endpoint works: agents do this every time. Keep the checks as a floor, and give them no points.
Leave questions open on purpose, and grade the question. “What happens after the last retry?” separated agents more than any functional requirement did, even with the brief telling them to record such calls. A candidate who asks, or states an assumption, is showing you judgment. One who silently picks an answer is showing you the agent’s default.
Plant one requirement that should be questioned. It was the cleanest signal in the experiment, and it tests the thing you are actually hiring for.
Put the critical fact outside the code. In the bug-hunt round, the only bugs that separated anyone lived in a business rule written in prose. Real work is like that; interviews rarely are. A domain rule in a README, a constraint mentioned only in conversation, a customer promise in a ticket: that is where a person’s reading and connecting still shows.
Check the claims, not just the code. On FeedMe, the most complete submissions came with the most overstated reports. Ask the candidate to walk you through what they built and what they tested, then check. That also happens to be the skill that reviewing agent output now depends on.

None of this bans AI. You couldn’t enforce a ban if you tried, and the job itself uses agents. It moves the interview to the parts of engineering that were always the point.

Limitations

Isolation took three rounds. The first 423 runs (every format except FeedMe) ran on my machine with environment-level isolation, before the incidents above, with Codex’s web search available. Their transcripts show no credential use, pushes, web searches or connector calls, but my credentials were within reach. The 15 FeedMe runs that count ran in the container. All 20 excluded runs are kept in the repository as evidence.
Credentials inside a run. Even in the container, an agent can read the Claude token and the Codex login (my full ChatGPT account) during its run. Scoring mounts every task in a format, so scored code could read other tasks’ answer keys, and a solution’s own tests run as the same user as the checker. The 423 host runs had no MCP lock; one resumed Claude session had my claude.ai connectors attached but called none of them. When those 423 were scored, the agents’ own test files ran on my machine.
Two criteria were invited. The webhook brief tells candidates to record their decisions and says time-dependent tests are weighed, so those criteria measure following that instruction, not unprompted judgment.
These are agents, not candidates. This measures what an agent does unaided, which is the floor a candidate’s own contribution has to clear. It does not measure how candidates use agents, or how interviewers score them.
The tasks and rubrics are mine. Rubrics were frozen before the first paid run and are public, but they reflect my idea of what matters. The authored tasks are small, and three repeats per cell (five for take-homes) show a pattern rather than pin rates.
The judge is a model. Every non-automated label comes from three Claude Opus votes. The blind hand-check was also done by a Claude agent, not a person, agreeing on 98 of 99.
FeedMe is public, with solutions online. That is part of the point: candidates can find them too. The authored tasks and custom questions are the control.
Cost: about $315 of Claude usage at API prices, most of it the judge, plus Codex on a subscription. Two judge batches’ costs were not recorded (one lost to a crash, one cut off by a usage limit) and are estimated.

Frequently Asked Questions

Can AI coding agents pass technical take-home assignments?

In this experiment, yes, completely on the automated side. Claude Sonnet 5.5, Claude Opus 5.5 and Codex met every automated correctness criterion on all three take-homes in every run, including a real company’s public assignment. What varied was judged: open questions, edge cases the brief didn’t state, honest reporting, and whether they questioned a bad requirement.

Should companies ban AI in technical interviews?

The data here argues for changing what is graded, not banning tools. In CoderPad’s 2026 survey, 34% of companies ban AI in assessments and 46% allow it. Bans are hard to enforce when a live round can be piped to a model in seconds. Grading judgment, questions asked and claims checked works whether or not the candidate used an agent.

Are LeetCode-style questions still useful for hiring?

Not for separating strong candidates who have an agent. Three current agents passed all 90 of their runs across known, modified and custom algorithm questions; only a smaller model failed any. 21 of 30 questions carried no signal at all.

What should a take-home test for in 2026?

The parts an agent doesn’t do on its own: asking about deliberately open questions, catching domain edge cases the brief omits, pushing back on a requirement that conflicts with the rest of the system, and reporting accurately what was built and tested. Put at least one critical fact outside the code, and grade the conversation about the work as much as the work.

Sources

All retrieved 2026-09-30.

Tristan Hume, Designing AI-resistant technical evaluations, Anthropic, 21 Jan 2026. https://anthropic.com/engineering/AI-resistant-technical-evaluations
Anthropic, original_performance_takehome. https://github.com/anthropics/original_performance_takehome
CoderPad, State of Tech Hiring 2026, 11 Feb 2026. https://coderpad.io/survey-reports/coderpad-state-of-tech-hiring-2026/
interviewing.io, How hard is it to cheat with ChatGPT in technical interviews? https://interviewing.io/blog/how-hard-is-it-to-cheat-with-chatgpt-in-technical-interviews
FeedMe, se-take-home-assignment. https://github.com/feedmepos/se-take-home-assignment
The harness, tasks, rubrics and every run: https://github.com/iceinvein/interview-signal