The Apprenticeship Was in the Comments

In January 2026, Anthropic sat 52 mostly-junior engineers in front of a Python library none of them had used. Half were allowed AI assistance, half were not. Afterwards both groups took a quiz with no AI available: debugging, code reading, code writing, conceptual understanding.

The AI-assisted group averaged 50%. The unassisted group averaged 67%. About two letter grades, Cohen’s d = 0.738, p = 0.01. The widest gap of any category was debugging.

Debugging is the skill the entire supervision model rests on. Every “human in the loop” argument assumes a human who can look at plausible-looking generated code and tell whether it is actually right. The assistance erodes hardest the exact capability its own oversight depends on.

And the place that capability used to be transmitted is closing. Last week I published a measurement across 12,291 merged pull requests: review discussion per 100 lines added fell 36% between 2025 and 2026, in the same direction in all five repositories. I read that as a quality finding at the time. It is also a teaching finding, and nobody is tracking it as one.

Key Takeaways

• Code review’s measured outcome was never mainly defect-finding. Bacchelli and Bird’s ICSE 2013 study at Microsoft found the expectation was defects and the realised outcomes were knowledge transfer, team awareness and alternative solutions.
• The channel that carries that is down 36% per line in my dataset, and 59.4% of 2026 merges carried no inline comment at all.
• Anthropic’s controlled study: AI-assisted juniors scored 50% against 67% on a post-task comprehension quiz, worst on debugging, while finishing only about two minutes faster (not statistically significant).
• Mandating an approver does not fix it. The one repository that drove unreviewed merges from 13.2% to 3.3% posted the steepest depth collapse of all five, 0.79 to 0.27 threads per 100 lines.
• The erosion is not uniform. In the same study, high scorers used AI to ask conceptual questions and request explanations; low scorers delegated generation and debugging. Constrain the mode of use, not the access.

What was code review actually for?

Not defects. Or rather: not mainly defects, however much teams say otherwise.

The reference point here is a 2013 study by Alberto Bacchelli and Christian Bird at Microsoft, which observed, interviewed and surveyed developers and managers and then manually classified hundreds of review comments. Its central result is a mismatch. Teams enter review expecting to find defects. What review actually produces, measured against those expectations, is less defect-finding than anticipated and considerably more knowledge transfer, team awareness, and generation of alternative solutions. The paper also finds that code and change understanding is the core activity, and that tooling largely fails to support it.

That result has been quoted for a decade as an argument for reviewing more carefully. It is worth reading the other way. Review was the one place in a normal week where someone who understood why this codebase is shaped the way it is had to write that down, addressed to a specific person, about a specific change, on a deadline.

Nothing else in the week is structurally like that. Design docs are written for the room, not for the newcomer. Pairing is expensive and irregular. Onboarding docs go stale in a quarter. Review was routine, mandatory, specific, and attached to work the junior actually cared about because it was theirs and it was blocked.

That is what an apprenticeship is: watching someone competent make a judgment about the thing in front of you, then being asked to make the next one. The industry got that for free as a side effect of a quality-control process, which is why nobody protected it. It never had a line item.

I made this argument narrowly once before, as a bullet in the case for AI as the first line of review: a bot cannot mentor. This post is that bullet with the measurement attached.

How much of that channel is left?

Roughly two thirds of it, and falling.

Across ruff, prisma, tailwindcss, vite and astro, review threads per 100 lines added went from 0.77 in 2025 to 0.50 in 2026. Down 36%, and down in five repositories out of five. The share of PRs merged with no review at all rose from 9.6% to 15.0%, and 59.4% of 2026 merges carried no inline comment anywhere on the diff.

The method, the confounds and the reproduction scripts are all in the original post, and the important caveat travels with the number: threads proxy discussion, not understanding. A brilliant three-line comment counts once. A pedantic argument about naming counts four times. In aggregate and across a year the direction is meaningful; per PR it tells you nothing.

The corroboration I did not have when I published is a July 2026 paper from CMU. Agarwal, Miller, Kästner and Vasilescu analysed more than 38,000 grey-literature documents about AI and code review, coded 3,100 of them, and built a causal model of 26 constructs and 67 relationships. They report that AI-authored pull requests are reviewed less often, merge faster and attract less discussion, while being explicit that the direction is sensitive to how the underlying data is sliced. Their framing is the one I would now borrow: review is the control point through which a coding agent’s effect on a codebase is decided.

They also name the thing this post is about. Their practitioners worry that AI adoption quietly erodes the understanding it once built. Two methods that share no data at all, mine mining repositories and theirs reading what practitioners say, arriving at the same place.

Doesn’t the AI reviewer fill the gap?

It fills the defect-finding half. The half that was never the main product is the half it does not touch.

Adoption is not the issue. AI code-review usage went from 14.8% in January 2025 to 51.4% in October across DORA’s cohort, and the commentary is often good: null checks, missed error paths, inconsistent naming, the mechanical layer that always ate the most reviewer minutes for the least judgment.

Four things it does not do, and none of them are model-quality problems that scale away.

It does not carry the history. The reason this module has an ugly seam in it is that a customer migration in 2023 needed both formats live for a quarter. Nobody wrote that down. The senior reviewer knows it, and the review comment is where they say it.

It does not calibrate to the reader. A good mentor pitches an explanation at the person receiving it and skips what they already know. A reviewer that files the same six findings on everyone’s PR is giving the junior and the principal identical treatment, which serves neither.

It is not accountable. The comment that carries weight is the one from someone who will be in the incident channel at 3am if it is wrong. That asymmetry is what makes a junior stop and reread rather than click resolve.

And nobody builds a relationship with it. The value of a review comment is partly that it makes the next question cheaper to ask. There is no next question with a bot; there is only the next PR.

I should be straight about my own position here. I published a working recipe for running your own AI reviewer on a Claude Code subscription, and I stand by it. What I have watched it do is raise the number of comments on a pull request while lowering the number of comments from a person who knows why the code is like this. Those two move in opposite directions and only one of them is visible.

The failure mode compounds when the reviewer is agreeable, which is the default. A model tuned to be pleasant will tell a junior their approach is reasonable, because it usually is, in isolation, on its own terms. A reviewer optimised to agree with you is the worst possible teacher, and we installed one in the teacher’s chair.

Why doesn’t requiring an approver fix it?

Because coverage and depth are different quantities, and only one of them appears on a dashboard.

My own dataset ran this experiment without meaning to. Four of the five repositories saw unreviewed merges rise. astro went the other way: 13.2% to 3.3%, stepping from 18.9% in 2025-Q3 to 4.1% in Q4 and staying there. That is not a change of heart, that is a branch-protection rule landing.

So astro reached roughly 97% review coverage. And astro posted the steepest depth loss in the dataset, 0.79 to 0.27 threads per 100 lines, a 66% fall and the only per-repo depth result significant on its own.

Requiring a review produces a review. It does not produce reading, and it certainly does not produce explaining. When every PR must carry an approval and the PRs keep getting bigger, the approval is what scales, because the approval is what is measured.

For the teaching argument this matters more than it does for the quality argument. A rubber stamp still stops some bad code, occasionally, by accident. A rubber stamp teaches nobody anything, ever, by construction. The governance report will show 97% and the apprenticeship will be at zero, and the two numbers are not in tension because they were never measuring the same thing. This is the same failure as an aggregate lead time that hides where the work moved: the metric survived, the meaning did not.

What does the assistance actually take from a junior?

Its ability to debug, first and worst. That is the finding, and it comes from an experiment rather than a correlation, which is why this study earns its place. Everyone had used Python weekly for at least a year and nobody had used the library. Two tasks, half the group assisted, then a quiz weighted towards debugging, code reading and conceptual problems.

Fifty against sixty-seven. The researchers’ own reading of the debugging result is the mechanism worth carrying: when you hand an error to a model, you skip the encounter-diagnose-resolve loop that debugging intuition is built out of. You get the fix and you do not get the practice. I would not push that further than the authors do, and they do not push it into neuroscience.

The part that should end the “but it makes them faster” reply: the AI group finished about two minutes quicker, and the difference was not statistically significant. In this study there was no throughput gain to weigh the comprehension loss against. There was just the comprehension loss.

Then put that next to what the same engineers are being hired to do. The plan, across most of the industry, is that AI writes more of the code and humans supervise it. Supervision means reading generated code and deciding whether it is right, which is debugging with extra steps. We are proposing to staff that with a cohort whose debugging formation the tool degrades most.

There is a general-population version of this result. Microsoft and CMU surveyed 319 knowledge workers about 936 real AI use cases and found that higher confidence in generative AI tracked with less critical thinking, while higher self-confidence tracked with more (Lee et al., CHI 2025). The variable is not the tool. It is whether the person still believes their own judgment is load-bearing, which is a thing a working environment either reinforces or does not.

This is the same instinct as knowing when to take the keyboard back and the same instinct as checking a model’s claims yourself. Those are habits. Habits are taught.

What separates the juniors who still learn?

How they ask. The same study answers this, and it is the most useful thing in it.

The erosion was not uniform. The high scorers in the AI group used AI. What they did with it was ask conceptual questions, request explanations, and follow up. The low scorers leaned on it to generate code and to do their debugging for them, which the researchers describe as cognitive offloading. Same tool, same task, opposite outcome, and the difference is a usage pattern rather than a talent.

That converts into rules a team can actually set. Five that follow directly from the evidence above:

The first debug attempt is unassisted. The agent may explain an error message. It may not fix the bug until the author has formed a hypothesis and said it out loud. This protects the exact loop the study identifies, and it costs minutes.

Route review by learning value, not only by risk. A junior’s pull request gets a named human reviewer even when the bot has already approved it, and especially then. Most routing today is risk-weighted, which sends senior attention to the scariest diff and leaves the newcomer with a bot.

Make the reviewer ask, not patch. A review prompt that asks the author why they chose this approach is worth more to a junior than a suggested diff they can accept without reading. This is a two-line change to a review prompt and it is the highest-leverage item on the list.

Instrument depth, segmented by tenure. Review threads per 100 changed lines, split by author seniority, quarter over quarter. Coverage will look fine. Depth for your least experienced engineers is the number that tells you whether anyone is still teaching, and nobody currently computes it.

Keep explanations in the pull request. An explanation given in a private chat window with a model teaches exactly one person and leaves no trace. The same explanation in a PR thread is durable, searchable, and read by everyone who touches that file later. Where the knowledge lands is a choice.

Two of these lean on machinery I have written about elsewhere: gating on trajectories rather than outputs frees human attention for the parts that teach, and the limits of writing your conventions down are a reminder that a document is not a substitute for someone explaining why the convention exists.

What breaks if this holds?

The supervision model assumes a continuing supply of engineers who can tell correct from plausible. That supply was manufactured, slowly and mostly invisibly, by the apprenticeship this post says is narrowing.

I am not going to attach dates to that. Anyone who tells you exactly which year the senior shortage arrives is guessing. The honest form of the claim is conditional. If review depth keeps falling and nothing replaces what it was carrying, the cohort learning to program now arrives at the years where we need their judgment with less of it than their predecessors had at the same point. Whether that binds depends on how many teams read the same evidence and change something.

Which is roughly where the CMU authors land too. Their causal model puts review at the control point and is explicit that AI does not determine the outcome on its own. Team expertise and the structure of the review process decide whether the effect is good or bad. The variable is not the model, and it was never going to be. It is what your team does at the gate.

FAQ

Does AI code review replace mentoring?

No. It replaces part of what review was doing, not the part that taught anyone. Bacchelli and Bird’s ICSE 2013 study found review’s realised outcomes were knowledge transfer, team awareness and alternative solutions rather than the defect-finding teams expected. Automated review addresses the defect half well and the knowledge half not at all, because it has no history with your codebase, no accountability for the outcome, and no relationship with the author.

Does using AI actually make developers worse at debugging?

In a controlled study, yes, for people learning something new. Anthropic gave 52 mostly-junior engineers an unfamiliar library, allowed half of them AI, and then quizzed both groups without it. The AI group averaged 50% against 67% (Cohen’s d = 0.738, p = 0.01), and debugging was the widest category gap. The AI group also finished only about two minutes faster, a difference that was not statistically significant.

Doesn’t requiring an approval on every pull request fix this?

It fixes coverage, which is not the same thing. In my dataset the one repository that drove unreviewed merges from 13.2% to 3.3%, consistent with a branch-protection rule, also posted the steepest fall in review discussion of all five, 0.79 to 0.27 threads per 100 lines. Mandated approvals scale; reading and explaining do not.

How do I let juniors use AI without this happening?

Constrain the mode of use rather than the access. In the same Anthropic study the high scorers used AI too: they asked conceptual questions and requested explanations. The low scorers delegated code generation and debugging wholesale. The practical version is a rule that the first debug attempt is unassisted and that the model explains errors rather than fixing them.

The number was doing two jobs

I am not going to end this by asking anyone to review more. Nobody has the hours, and the whole finding of the 12,291-PR dataset is that nobody chose this. Load rose 42% a month, latency did not move, and review depth absorbed the difference quietly. Every individual reviewer behaved reasonably.

What is worth noticing is narrower. One activity was doing two jobs. We automated the job it was worst at, kept the metric that tracks the job we automated, and lost the job it was actually best at without ever putting a number on it. Coverage is on every dashboard. Discussion per 100 lines is on none of them, and that is the one that measures whether anybody is still learning anything.

Which leaves the uncomfortable version. The plan is that humans supervise the machines. Supervision is debugging. Debugging is what the assistance takes first, and review discussion is where it used to be handed down.