Your Agent Recovers From Everything It Can See
In March 2026, six researchers published the first systematic study of what actually breaks in AI coding tools. They manually classified more than 3,800 publicly reported bugs across the open-source repositories of Claude Code, Codex CLI and Gemini CLI, using open-card sorting so the categories emerged from the data rather than being imposed on it.
The headline is a rebuke to most of the discourse. API and integration errors were the single largest root cause at 21.4%. Configuration and setup came second at 15.9%. Environment and platform compatibility, 10.5%. Command execution and parsing, 6.8%. And the category everyone argues about, AI logic and behaviour, came in at 10.0%. Their full root-cause taxonomy runs to ten categories over 3,864 bugs, split Claude Code 2,343, Codex CLI 1,192, Gemini CLI 329.
Their conclusion, in their words, is that building reliable AI coding tools is fundamentally a systems engineering problem. Not a model problem.
That result got picked up as proof that agents are brittle. I wanted to check it against something the paper cannot see, because a reported GitHub bug is a failure someone was annoyed enough to write up. So I went to my own transcripts.
Evidence for everything below: every Claude Code session on this machine, 1,278 session transcripts across 13 project directories, snapshotted 2026-09-09T00:59Z with the analysis session itself excluded. 81,123 tool results, of which 2,045 returned an error. User denials are excluded from both numerator and denominator, because a human saying no is not a failure. Classification rules, confounds and the full residual bucket are at the bottom.
Key Takeaways
- 2.52% of tool calls fail. Of those failures, 93.1% are harness, environment or policy. Not one classified failure is model reasoning, and that is a property of the channel rather than a compliment to the model.
- The brittleness framing does not survive the recovery data. Mean consecutive-failure run is 1.09, 92.1% of failure runs are a single call, the agent succeeds on the very next call 91.2% of the time, and the longest streak in 1,278 sessions is four.
- The error channel structurally cannot report a reasoning error. A bad decision returns a successful tool call with a wrong result. Every bug taxonomy finds plumbing partly because plumbing is the only thing that raises a flag.
- The largest single category, 38.1%, is a command that exited non-zero while naming no cause at all. 693 of those 779 are a bare
exit 1, and 21.4% carry almost no text beyond the exit code.- The second largest, 26.1%, is my own policy layer. 495 of 534 are one hook I wrote. Free recovery is what let me ship a hook that fires wrongly hundreds of times without noticing.
What do agent failures actually look like?
Overwhelmingly, they look like a shell command that did not work.
I classified every erroring tool result with ordered regex rules, first match wins, and kept an explicit unclassified bucket rather than forcing everything into a category. Here is the split.
Group the first three plus the small integration and config buckets and you get 1,904 of 2,045, or 93.1% harness, environment or policy. The same four categories in their taxonomy (API and integration, configuration, environment, command execution) sum to 54.6%. My number is higher, and the reason is not that my setup is worse. It is that their corpus contains bug reports about the tool, including usability and UI complaints that have no analogue inside a session, while mine contains only what the runtime actually flagged.
The distribution across tools is what you would guess once you see it.
Bash is 81.6% of every tool call I make and 91.2% of every failure. Structured tools barely fail: Write at 0.44%, SendMessage at 0.00% across 316 calls. The difference is not that writing files is easier than running commands. It is that a structured tool has a schema, a bounded set of outcomes and an error string written by whoever designed it, while a shell command has an integer and whatever the program felt like printing. I have argued before that the tool description is the prompt your agent reads. The failure data says the tool’s error contract is the other half of that, and almost nobody designs one.
The rate is also stable across projects, which matters because a single pathological repository could produce all of this. It does not. Five projects with 500 or more calls each land between 1.42% and 2.81%, and the pooled subagent transcripts sit at 2.51%. Different languages, different stacks, same band.
Doesn’t that mean agents are brittle?
No, and this is where I stopped believing the framing I started with.
A 2.52% failure rate sounds alarming until you ask what it costs. So I measured what happens on the call immediately after a failure, and the distribution of consecutive-failure runs.
Mean consecutive-failure run: 1.09 calls. The agent succeeds on the very next call 91.2% of the time. 92.1% of failure runs are one call long. The longest run anywhere in 1,278 sessions is four, and there are five of those.
There is no death spiral in this data. There is no thrashing. The wrong working directory, the missing binary, the blocked command, the bare exit 1: the model reads the error, adjusts and moves on, almost always immediately. Whatever brittleness means, it does not mean this.
Which forces a harder question about what the taxonomies are measuring.
So what does the error channel not see?
Everything that matters most.
A tool call raises is_error when the mechanism fails. The file was not there, the command exited non-zero, a hook blocked it, the socket refused. Every one of those is a fact about the world that the runtime can check.
Now consider the failure everyone actually worries about. The agent reads the wrong file and is confident about it. It greps a pattern that silently matches nothing and concludes the code does not exist. It writes a test that passes because it asserts nothing. It edits the correct-looking function in the wrong module. Every one of those returns is_error: false. A successful tool call with a wrong result is indistinguishable, at the channel level, from a successful tool call with a right one.
So the finding is not that agents fail on plumbing. It is that plumbing is the only thing the error channel can express, and the reason every bug taxonomy converges on plumbing is partly that plumbing is the only failure class that raises its hand. The FSE paper’s 10.0% AI-logic slice is not a measurement of how often models reason badly. It is a measurement of how often a human noticed and filed an issue about it.
That reframes the recovery number too. A 91.2% next-call recovery rate is not evidence that the agent handles failure well in general. It is evidence that the agent handles legible failure well. The failures it recovers from are the ones that come with a diagnostic. The failures that cost you a day come with a green checkmark.
There is a worse case than the agent not noticing. In a later experiment where Claude Code wrote tests for functions with planted bugs, Opus 5.5 named the bug in its closing message in 35 of 54 runs but wrote a failing test for it in only 14. In the other 21 it saw the problem, said so in prose, and shipped a green suite that encodes it. The diagnosis existed. It went to the one channel no pipeline reads.
This is the same structural problem I hit when Codex encrypted its sub-agent payloads and the readable audit trail went with it. Observability that only records what the runtime already knew how to describe is not observability of the system, it is observability of the instrumentation.
Why does a third of the data have no cause at all?
The single largest category, 779 failures, is a command that exited non-zero while naming no recognisable cause. That is not a rounding bucket. It is the biggest thing in the dataset, and it is worth looking at closely.
Of those 779, 693 are a bare exit 1. The rest are a long tail: 28 exit 2, 28 exit 143, 14 exit 128, five exit 7.
The tail is the interesting part, because those codes carry real information. Bash’s convention reserves 128 plus the signal number for a fatal signal, so 143 is 128+15, a SIGTERM, usually my own timeout. 127 means the command was not found, 126 that it was found but not executable. The manual is explicit that non-zero was chosen over a single failure value so there would be “a variety of ways to indicate various failure modes”. The vocabulary exists. Almost nothing uses it.
Split by what text came back with them:
- 42.6% returned program output with no error text at all. Something printed normally, then the process exited non-zero.
- 35.9% named a failure (“error”, “failed”, “cannot”) without naming a cause.
- 21.4% were near-empty. An exit code and essentially nothing else.
So roughly one in five of the largest failure category hands the agent an integer and no information. The model then has to reconstruct what went wrong from the command it just ran, which it does successfully most of the time, and that success is precisely what makes the problem invisible.
This is a fixable engineering defect and it does not live in the model. It lives in the scripts, Makefiles, test runners and CLIs the agent shells into. Exit 1 is what a program returns when nobody decided what it should return, which is most programs, because until recently the thing reading the failure was a human who could also read the screen. If you want your agent to spend fewer tokens guessing, the highest-leverage change available to you is making your own tooling say why it failed on stderr. That is a Makefile edit, not a prompt.
It also explains why Bash fails 6.4 times more often than Write. Structured tools return errors somebody designed. Shell commands return errors nobody designed.
Who is causing these failures?
Here is the number I did not enjoy finding.
Of 534 permissions and policy failures, 495 came from one hook I wrote myself. It is a PreToolUse hook that intercepts grep and find in indexed repositories and tells the agent to use my code-intelligence server instead. Thirty-three more are approval gates on multi-part shell commands. Six are actual operating-system permission errors. Those three account for all 534.
So 24.2% of every failure in this entire corpus is my own policy layer blocking my own agent, in a repository where I configured it to do exactly that.
Some of that is the hook working as designed: it fires, the agent redirects to the right tool, everyone wins. But 495 firings is not a guardrail catching rare mistakes, it is a guardrail sitting in the default path. A rule that fires hundreds of times is not a rule the agent has learned; it is a rule the agent keeps rediscovering, one blocked call at a time, and paying a round trip for each rediscovery.
I did not notice for months, and the recovery data explains why. Every one of those 495 blocks was absorbed on the next call. Free recovery is exactly the condition under which a badly-placed policy can run for a very long time without anyone measuring it. Hooks are the only deterministic substrate Claude Code gives you, which is what makes them worth using and also what makes a misconfigured one so quiet. The fix is not to remove the hook. It is to move the instruction where it is read before the call rather than after it, and to treat a high firing rate as a defect report about my own configuration.
The environment bucket has a smaller version of the same story: 431 of 545 are a missing file or directory, overwhelmingly a command run from the wrong working directory. Not a model failing to understand paths. A harness that lets a subshell’s cd evaporate between calls.
What should you instrument?
Four numbers, all computable from transcripts you already have on disk, none requiring new tooling.
Tool-call failure rate, split by tool. The aggregate is nearly useless because Bash dominates it. The split is where the signal is. If a structured tool climbs above about 1%, its schema or its error contract is wrong.
Hook and policy firing rate, as a share of failures. This is the one nobody computes and the one that found a real defect in my setup. Any single policy responsible for more than a few percent of your failures is misplaced, not protective.
Share of failures with no attributable cause. Mine is 38.1%. Track it against your own tooling, because unlike everything else on this list you control the fix directly. It goes down when your scripts learn to write to stderr.
Mean consecutive-failure run length. The cheap health check. At 1.09 the plumbing is noise. If it climbs past 2, something has genuinely broken and it will show up here before it shows up in your patience. This is the number that belongs in an agent oncall runbook as a threshold.
What none of these measure is the thing this post says is invisible. For that you need a different instrument entirely: assertions over what the agent did, not over whether the call succeeded. That is gating on the trajectory rather than the output, and it is the only approach I know of that catches a failure wearing a green checkmark.
What I got wrong, and what this cannot tell you
I started this expecting to replicate a brittleness finding and write about hardening the harness. The taxonomy replicated cleanly and the conclusion inverted, because I measured cost as well as frequency. If I had stopped at the distribution I would have published a more dramatic and less true post.
The comparison to the FSE study is looser than it looks and I want to be explicit about it. They classified reported bugs in the tools. I classified runtime failures inside sessions. Different populations, different units, and their categories include things like usability and UI that have no in-session equivalent. Both find that plumbing dominates, and that agreement is worth something, but do not read 93.1% and 52.8% as measurements of the same quantity.
The corpus is one engineer on one machine. My hook configuration produced a quarter of my failures, which is a fact about me, not about Claude Code. Somebody with no hooks would see that 26.1% vanish and every other share rise proportionally. The shape is probably general. The specific percentages are mine.
Classification is regex over error text, ordered, first match wins. It is transparent and it is crude. The 5.3% unclassified bucket is real and I have left it in rather than tuning it away: it is mostly schema-validation errors from my own subagents, worktree guards, and approval prompts phrased in ways my rules do not catch.
And the central claim, that the error channel cannot express reasoning failures, is an argument from the structure of the data rather than a measurement. I cannot count the failures I cannot see. That is the whole point, and it is also the limitation.
FAQ
Are AI coding agents unreliable?
Not in the way the bug taxonomies suggest. In 81,123 tool calls across 1,278 sessions, 2.52% failed, and the agent recovered on the very next call 91.2% of the time with a mean failure run of 1.09 calls. The mechanical reliability is high. The open question is not whether tool calls succeed, it is whether successful tool calls did the right thing, and no error rate can answer that.
What actually causes AI coding agent failures?
Plumbing. Across my corpus, 93.1% of failures were harness, environment or policy: a command exiting non-zero with no stated cause (38.1%), a missing file or wrong working directory (26.7%), or a permission or hook block (26.1%). The FSE 2026 study of 3,864 reported bugs in Claude Code, Codex CLI and Gemini CLI reached the same conclusion from a different direction, putting AI logic and behaviour at 10.0% of root causes.
Why does Bash fail more than other agent tools?
Because shell commands have no error contract. Bash failed on 2.82% of 66,228 calls while Write failed on 0.44% and SendMessage on 0.00% of 316. A structured tool returns an error string someone designed for a reader; a shell command returns an integer and whatever the program happened to print. 693 of my 779 unattributed failures were a bare exit 1.
Should I add more retries to my agent harness?
Almost certainly not. Retries address a problem this data does not show: 92.1% of failure runs are already a single call and nothing in 1,278 sessions ran longer than four. The model is doing the retry, adaptively, using the error text. The higher-leverage work is making your own scripts explain themselves on stderr, so the adaptation is informed rather than guessed.
How do I catch agent mistakes that do not raise an error?
Not through the error channel, which structurally cannot report them. A wrong file read, an empty grep treated as evidence of absence, or a test that asserts nothing all return success. Catching those needs assertions over the agent’s trajectory and over observable outcomes, run as gates, rather than monitoring of tool-call exit status.
Methodology
Data. Every Claude Code session transcript under ~/.claude/projects on one machine, 1,278 JSONL files across 13 top-level project directories, snapshotted at 2026-09-09T00:59Z. The transcript for the session that performed this analysis is excluded, because the corpus is append-only and live while being read. Unit of analysis is the tool_result content block: 81,123 total, 2,045 with is_error set.
Exclusions. User denials (“the user doesn’t want to proceed”, “tool use was rejected”, plan-mode captures) are removed from both numerator and denominator. There were 11. A human declining a proposed action is not a tool failure and counting it as one inflates the rate.
Classification. Eight ordered regular expressions over the error text, first match wins, with everything unmatched recorded as unclassified rather than redistributed. Order matters: a message reading Exit code 1 | sed: no such file is classified as environment rather than unattributed, because the environment rules run first. The residual command_failed_unattributed category is therefore genuinely residual: an exit code with no cause any rule could recognise.
Recovery. Failure runs are maximal sequences of consecutive erroring tool_result blocks in transcript order within a session, denials removed. “Recovered” means the next tool_result in that session existed and did not error. The six failures that ended a session therefore count as non-recoveries, which is the conservative direction: excluding them would raise the recovery rate rather than lower it.
Confounds, in order of how much they should worry you.
This is one engineer, one machine, one configuration. The hook that produced 24.2% of all failures is mine. Share figures are not portable; the shape probably is.
Regex classification is crude. It reads error text, not intent. A message can name two causes and gets attributed to whichever rule runs first. I chose transparency over accuracy and published the rule order so the choice is auditable.
is_error is the runtime’s judgement, not ground truth. Some tools return a soft failure with is_error unset, and those are invisible here in exactly the way the post argues everything interesting is invisible here.
Sessions are not independent. Long sessions contribute more calls, and a single misconfigured afternoon can move a category. The per-project spread (1.42% to 2.81% across five projects with 500 or more calls) is the check on this, and it is reassuring rather than conclusive.
Subagent transcripts are pooled. 48,612 of the 81,123 calls come from subagent sessions, which are not attributable to a single project directory. Their 2.51% rate sits inside the band of the named projects.
Reproduction. The whole classification and recovery analysis is one file. It reads only ~/.claude/projects, writes nothing, and needs no dependencies. Pass your own session id as an argument to exclude the session you run it from.
import collections, glob, json, os, re, sys
DENIAL = re.compile(r"user doesn't want to proceed|tool use was rejected|"
r"client captured your proposed plan", re.I)
RULES = [ # ordered, first match wins; order decides an error naming two causes
("permissions_policy", r"hook error|PreToolUse|PostToolUse|<tool_use_error>Blocked|"
r"requested permissions|permission denied|Operation not permitted|EACCES"),
("api_integration", r"curl: \(\d+\)|Failed to connect|Connection refused|ECONNREFUSED|"
r"ETIMEDOUT|ENOTFOUND|50[234] |rate limit|429|socket hang up|EPIPE|fetch failed|"
r"not bound to a repo|MCP"),
("environment_platform", r"No such file or directory|does not exist|cannot find|not found|"
r"FileNotFoundError|ENOENT|MODULE_NOT_FOUND|command not found|is not recognized|"
r"is not installed|EISDIR|ENOTDIR"),
("state_context", r"has been modified since|String to replace not found|has not been read yet|"
r"file has been unexpectedly|old_string|not unique|InputValidationError|"
r"could not be parsed as JSON"),
("config_setup", r"missing .{0,20}(config|env|key|token|credential)|is not set|"
r"not configured|invalid configuration|API key|unauthorized|401|403|authentication"),
("resource_limits", r"out of memory|OOM|heap|maximum call stack|too large|exceeds maximum|"
r"ENOSPC|token limit|context (window|length)"),
("command_failed_unattributed", r"Exit code \d+|\(eval\):|zsh:|bash:|sh: "),
("syntax_parse", r"syntax error|unexpected token|SyntaxError|parse error"),
]
RULES = [(name, re.compile(rx, re.I)) for name, rx in RULES]
cat, streaks, after = collections.Counter(), collections.Counter(), collections.Counter()
total = failures = 0
for path in glob.glob(os.path.expanduser("~/.claude/projects/**/*.jsonl"), recursive=True):
if len(sys.argv) > 1 and sys.argv[1] in path: # skip the analysing session
continue
seq = []
for line in open(path, errors="ignore"):
try:
blocks = json.loads(line).get("message", {}).get("content")
except ValueError:
continue
for b in blocks if isinstance(blocks, list) else []:
if not isinstance(b, dict) or b.get("type") != "tool_result":
continue
text = b.get("content")
if isinstance(text, list) and text and isinstance(text[0], dict):
text = text[0].get("text", "")
text = text if isinstance(text, str) else str(text)
if b.get("is_error") and DENIAL.search(text):
continue # a human saying no is not a failure
total += 1
seq.append(bool(b.get("is_error")))
if b.get("is_error"):
failures += 1
cat[next((n for n, rx in RULES if rx.search(text)), "unclassified")] += 1
run = 0 # consecutive-failure runs, transcript order
for i, errored in enumerate(seq):
if errored:
run += 1
if i + 1 < len(seq) and not seq[i + 1]:
after["recovered"] += 1 # a failure ending a session is not a recovery
elif run:
streaks[run] += 1
run = 0
if run:
streaks[run] += 1
print(f"{total} tool calls, {failures} failures, {failures / total * 100:.2f}%")
for name, n in cat.most_common():
print(f" {name:28} {n:5} {n / failures * 100:5.1f}%")
runs = sum(streaks.values())
print(f"mean failure run {sum(k * v for k, v in streaks.items()) / runs:.2f}, longest {max(streaks)}")
print(f"recovered on next call {after['recovered'] / failures * 100:.1f}%")Run against my corpus today it prints a rate of 2.54% rather than the 2.52% quoted above, because the corpus is append-only and has grown since the snapshot. The category shares move by less than half a point. The interesting version of this is not my corpus, it is yours, because the hook firing 495 times is going to be a different hook on your machine.
The paper says building reliable agents is a systems engineering problem, and it is right. What the session data adds is that the systems doing the failing are mostly ones you already own and can already fix, and that the reason nobody fixes them is that the agent quietly absorbs the cost on the very next call. The failures you can see are cheap and they are not the ones to worry about. The expensive ones return is_error: false.
If it was useful, pass it along.
