The Router Pattern: When to Downshift the Model

On 2026-05-26, OpenRouter raised a $113M Series B led by CapitalG at a roughly $1.3B post-money valuation, double what it was a year earlier, and disclosed that it now moves around 100 trillion tokens a month, up about 5x in six months, across 8 million-plus users and 400-plus models (TechCrunch, 2026). A routing layer is now a billion-dollar category. The reason is mundane: most of those tokens do not need the model they are being sent to.

That is the whole argument for routing, and it is arithmetic, not opinion. A 5-25x price spread separates the cheap tier from the top tier. Inside an agent pipeline, most turns are classification, extraction, and tool-routing, not synthesis. So the win is not “pick a better model for my session.” It is per-call: downshift the turns that don’t need the top model, and know the four places where doing so quietly breaks a coding agent.

Be clear about the scope, because the dominant search results aren’t. This is per-step routing inside a pipeline, where each subtask gets the cheapest model that clears its bar. It is not chatbot model selection, and it is not a roundup of hosted router products. It is also not agent-to-agent coordination: routing turns inside one pipeline is orchestration you own end to end, not delegation across separately-owned agents. The GPT-5 story (next section) sets up the tension: automatic routing is real and valuable, and it erodes trust the moment it goes silent.

Key Takeaways

A 5-25x price gap separates Anthropic’s cheap and top tiers (Haiku ~$1/$5, Sonnet ~$3/$15, Opus ~$5/$25 per million input/output tokens; platform.claude.com, 2026), so most agent turns overpay.

The router pattern: establish the quality bar with Sonnet first, then ask whether Haiku matches it. Default to Sonnet, downshift to Haiku for well-defined checkable subtasks, escalate to Opus for synthesis and ambiguity.

RouteLLM showed >85% cost reduction at 95% of GPT-4 quality on MT-Bench back in 2024 (LMSYS, 2024); but when I metered my own agent usage, auto-routing realized only ~7.6% versus all-Opus, because the un-downshifted Opus main loop was 93% of the bill.

Four steps punish a downshift and all fail silently: tool selection, long reasoning chains, messy-code edits, and the router call itself. Track quality per tier, not just blended cost.

Routing is now contested, not settled (the GPT-5 lesson)

Automatic per-prompt routing is real and valuable, and opaque routing erodes trust; GPT-5 proved both at once. When OpenAI shipped GPT-5 in August 2025, it included a real-time router that auto-selected between a fast model and a deeper reasoning model on every prompt, and the backlash was immediate: a “GPT-5 is horrible” thread drew more than 2,000 comments, and within days OpenAI re-added manual Auto, Fast, and Thinking modes and brought back GPT-4o (The Decoder, 2025).

Sam Altman’s framing is the lesson in one sentence: “most users will want Auto, but the additional control will be useful for some people” (The Decoder, 2025). Routing as a default is fine. Routing you can’t see or override is not.

For agent builders, the takeaway is sharper than “users like control.” A silent router that picks wrong is worse than no router, because you can’t tell why the output got worse. Did the model change? Did the prompt drift? Did a tool fail? When routing is invisible, a quality regression looks like model noise, and you’ll spend a week chasing a ghost. Routing has to be observable and overridable from the day you turn it on. Hold that thought; it’s the operating discipline the last section comes back to.

The 5-25x spread is the whole argument

The case for routing is arithmetic before it is anything else. Anthropic’s tiers span 5x on each axis, and the cross-tier gap is wider still: Haiku 4.5 runs about $1 input / $5 output per million tokens, Sonnet 4.6 about $3 / $15, and Opus 4.8 about $5 / $25 (platform.claude.com, 2026). Opus output is 5x Haiku output; Opus output against Haiku input is 25x. Send a classification turn to Opus and you are paying up to 25 times the floor for an answer Haiku would have nailed.

Two levers stack on top of routing, and they matter because tier selection is the one that actually moves the bill. Prompt caching cuts cached input by roughly 90% (cache reads bill at about a tenth of base input), and batch processing is 50% off (platform.claude.com, 2026). Caching stacks with a 90%+ cache hit rate on a stable prefix, and it composes with routing rather than competing with it. But squeezing the output itself has a ceiling: when I tried to cut Claude’s output tokens directly, the gains hit a wall around 5%. You can shave a little off how much a model says. You can change which tier says it by 5-25x. Tier selection is the lever.

Routing works, and the evidence is from 2024

The “downshift most calls” claim is not new optimism; it was demonstrated two years ago. RouteLLM, from LMSYS, hit 95% of GPT-4 performance using only 14% of GPT-4 calls on MT-Bench, an overall cost reduction north of 85%, with smaller but real cuts of 45% on MMLU and 35% on GSM8K, and its open routers matched commercial products like Martian and Unify while running over 40% cheaper (LMSYS, 2024).

Be precise about that 14%, because the precision is the point. It is the judge-augmented variant; the router trained on Arena data alone needed about 26% of GPT-4 calls to clear the same bar. Both are large wins. Quoting the 14% without the caveat is how a real result turns into a marketing number, and marketing numbers are exactly what this post is trying to replace.

Speaking of which: vendor and practitioner write-ups cluster savings around 50-80% on mixed workloads, climbing toward ~86% when something like 90% of traffic hits a nano tier. Treat that as a labelled practitioner range, not an authoritative claim. Routing works; the honest number is the one you measure on your own pipeline against an all-top-tier baseline, which is exactly what the next section does. The open question was never “does routing work.” It is “which calls,” and that is the rest of this post.

The decision tree: which calls to downshift

Default to Sonnet, downshift to Haiku only when the subtask is well-defined and verifiable, and escalate to Opus only on ambiguity or cross-file synthesis. The rule that keeps you honest is directional: establish the bar with Sonnet first, then ask “does Haiku match it?” Never start at Haiku and hope. Anthropic’s own model-selection guidance lands in the same place: test with the workhorse, then check whether the cheaper tier holds.

Establishing the bar is a measurement, not a vibe. Take twenty to forty real instances of the subtask, run them on Sonnet, and treat that as the reference output. Then run the same instances on Haiku and compare against a fixed rubric instead of eyeballing. If Haiku matches on the cases that matter, downshift it; if it misses on a recognizable slice, you have found exactly where the downshift breaks, before it ships rather than after a user does. The reverse order, starting cheap and trusting it, is how a quiet regression gets into production wearing a green dashboard.

That tree is what you should do. Here is what I was actually doing. I totalled the real per-call token usage by model across my own code-intelligence MCP work, 17,087 billed model calls over 3,115 agent sessions, and priced it against an all-Opus baseline.

Tier	Billed calls	Realized cost	If it had all been Opus
Haiku	847	$32	$159
Sonnet	2,955	$150	$250
Opus	13,285	$2,600	$2,600
Total	17,087	$2,782	$3,010

Realized savings: 7.6%. Not the 50-80% the vendor blogs advertise, not even comfortably into double digits: 7.6%. The reason is the third row: Opus was 93% of my spend. The cheap-tier work, the Explore subagents and other offloaded turns, was real but tiny next to the Opus main loop, which I had never deliberately downshifted. I was running the exact anti-pattern this post warns about, and the meter proves it: the realized number is set by how much of the expensive tier you actually move, and I had moved almost none of it.

That is the honest shape of routing. The 50-80% practitioner range is real, but it describes a pipeline that has been deliberately re-tiered, where classification, extraction, and ranking were pushed down on purpose. Auto-routing alone does not get you there; it offloads a few subagent turns and leaves the synthesis-heavy main loop exactly where it was. The floor on your savings is whatever you leave on the top tier, and by default that is most of it. Routing the model is also a different lever from routing the agent, where spawning subagents can burn an order of magnitude more tokens; the two compound, so a pipeline that fans out heavily has even more of its bill sitting on tiers it never examined.

Closing that gap is less exotic than it sounds, and the router does not have to be a model. A plain heuristic in the orchestrator routes the easy cases for free: candidate count, file count, whether the question names a single symbol or spans a subsystem. A model-based router buys flexibility on inputs you can’t characterize; a heuristic router buys a free, debuggable route on the inputs you can. Most pipelines want both, and the cheapest correct router is the one you don’t have to call.

Where downshifting silently degrades a coding agent

Four steps look cheap and punish a downshift, and all four fail the same way: degraded output, no error. That is what makes them dangerous. A crash you notice; a slightly-worse plan you ship.

Tool selection and argument construction. This is the riskiest downshift in the whole pipeline, and it is the one cost guides recommend most because it “looks simple.” A 2026 preprint reports that routing accuracy drops non-linearly with tool count, with a roughly 25-tool agent falling toward ~70% routing accuracy (arXiv:2603.01548, 2026, preprint). A 70% per-step accuracy compounds fast across a multi-step workflow: five tool calls at 70% each is a 0.7^5 ≈ 17% chance of getting the whole chain right. Downshifting the model that holds tool choice is how you turn a capable agent into a confidently wrong one.
Long reasoning chains. Long chains degrade small-model instruction-following. The cheap model starts on-plan and wanders off it three steps in, dropping a constraint it was holding at the start. The output still looks plausible, which is the trap.
Messy code. Clean, well-structured code yields to cheap models; messy, legacy, or under-documented code needs frontier reasoning to navigate. A 2026 preprint finds that code-quality signals predict the cost-effective tier for a software task (arXiv:2604.07494, 2026, preprint), which is a useful routing input: the worse the code, the higher the tier.
The router itself. “The router is a model too.” The classification call that decides routing costs tokens and can be wrong, and on short tasks that overhead can exceed the savings. Worse, a wrong route costs double: you pay for the cheap attempt, it fails a quality gate, and you pay again for the escalation. A misroute is more expensive than no routing at all.

Trap #1 is also why the main loop stays on Opus. Tool selection is the cut that looks cheapest and bites hardest: downshift the step that builds tool arguments and the failure is not a crash, it is the agent calling the right tool with slightly wrong arguments, a path scoped one directory too narrow, a symbol name truncated. The answer comes back confident and incomplete, and because nothing errors, a cost dashboard stays green while answer quality quietly slides. The defense is not a model change, it is a validation gate: a cheap structural check on the constructed call (does the path exist, does the symbol resolve) that escalates to a stronger tier on a miss. The gate costs a few tokens; the silent degradation costs trust in the whole pipeline, which is exactly the kind of regression a blended-cost number hides and a per-tier success metric catches.

Cascade vs. up-front classify: two ways to route, two different bills

There are two implementation patterns, and they fail in different directions. Up-front classification picks the tier before the call: a small model (or a heuristic) reads the task and routes it. A cascade tries the cheap model first and escalates on low confidence. Up-front classify pays the router tax on every call but routes once; cascade pays nothing up front but pays twice on the unhappy path, eating the cheap attempt plus the escalation.

Choose by your escalation rate. If a meaningful fraction of cheap attempts fail their quality gate, classify up front, so you stop paying for doomed first tries. If escalations are rare and your quality bar is strict, cascade, so the common case stays cheap and only the hard tasks pay the premium. One thing is not optional in a cascade: the confidence check that triggers escalation. Without a real gate, a cascade is just “use the cheap model and hope,” which is the failure mode from the trap section wearing a fancier name.

Put numbers on it. Say a cheap attempt costs $0.01, an Opus escalation costs $0.15, and your quality gate rejects one cheap attempt in three. A cascade pays the $0.01 every time and the $0.15 on the third that escalate, about $0.055 a call. Up-front classification pays a roughly $0.005 router tax on every call and then routes once, sending the same one-in-three to Opus, about $0.057 a call. That is nearly a wash at a 33% escalation rate. And that wash is the decision boundary: push the rate higher and the cascade’s pay-twice penalty wins, so you classify up front; pull it lower and the cascade’s cheaper happy path wins. Measure your real escalation rate before you pick; intuition is usually wrong about which side of the line you’re on.

This is also where you see the difference. The Agent SDK meter is where the per-call cost actually shows up, so you can compare a cascade’s escalation tax against an up-front router’s per-call tax with real numbers instead of intuition. If you’re on Claude Code, the 5-hour-window usage dashboard surfaces which tier is eating the budget, which is the input you need to decide whether your escalation rate justifies cascading at all.

Make routing observable, or do not route

Routing without per-call attribution is how you discover a regression three weeks late. The discipline is simple and non-negotiable: log the tier per call, the route reason, and the escalation rate, and alert when blended cost or task-success drifts. Track quality per tier, not just blended cost, because a cost-only view hides the exact failure routing introduces, a sharp success-rate drop on one downshifted step while the average looks fine. A 2026 preprint on multi-agent failures makes the same point: quality-aware routing matters precisely because optimizing on cost alone misses the success-rate cliff (arXiv:2603.04445, 2026, preprint).

This is the GPT-5 lesson applied to your own stack: routing must be observable and overridable. Every routed call should carry a tier tag and a route reason, so when output quality moves you can answer “which tier, and why” in one query instead of one week. Concretely, that is three fields on every call: the tier that ran, the reason it was chosen (which route rule fired, or which confidence check triggered an escalation), and whether the quality gate passed. With those, a regression hunt is “group by tier, plot success rate over the last week,” and the alert writes itself: page me when any tier’s success rate drops more than a few points, or when blended cost per task drifts outside its band.

Overridable matters as much as observable: keep a manual pin so you can force a route while you debug, exactly the Auto-or-Fast escape hatch OpenAI had to add back. The per-call lever is routing; the way the team sees whether it’s working is the three-axis, two-cap token budget that makes spend legible across everyone’s agents at once. Routing you can’t measure is just a new way to be wrong faster.

FAQ

When should an agent use a cheaper model instead of the top model?

When the subtask is well-defined with a checkable output: classification, extraction, tagging, or a templated summary. Default to Sonnet, downshift to Haiku for those well-scoped steps, and reserve Opus for synthesis and ambiguous multi-file reasoning. The directional rule matters: establish the quality bar with Sonnet first, then ask whether Haiku still clears it, never the reverse.

Does routing to a cheaper model degrade a coding agent?

It can, silently. The four high-risk downshifts are tool selection, long reasoning chains, messy-code edits, and the router call itself, and all four fail with degraded output and no error. A roughly 25-tool agent can fall toward 70% tool-routing accuracy (arXiv:2603.01548, 2026, preprint), and that error compounds across steps. The defense is per-tier quality tracking plus a validation gate, not blended cost alone.

How much can per-call model routing actually save?

RouteLLM showed over 85% cost reduction at 95% of GPT-4 quality on MT-Bench back in 2024 (LMSYS, 2024), and practitioner reports cluster at 50-80% on mixed workloads. The honest answer is to measure your own blended cost against an all-top-tier baseline. When I metered my own agent usage it was only ~7.6%, because 93% of the spend sat on an Opus main loop I had never deliberately downshifted; the 50-80% wins require pushing those turns down on purpose.

What is the difference between a model cascade and up-front routing?

Up-front routing classifies the task and picks a tier before calling. A cascade tries the cheap model first and escalates on low confidence. Cascade protects quality but pays twice on every escalation (the cheap attempt plus the retry), while up-front classify pays a small router tax on every call but routes once. Pick by your escalation rate: high escalation favors up-front, low escalation with strict quality bars favors cascade.

The lever that actually moves spend

The spread is 5-25x, most turns don’t need the top model, and the win is per-call. A billion-dollar routing category exists because most tokens are still sent to the wrong tier, and the fix inside your own pipeline is not a better model, it is a deliberate choice about which tier answers each call. The same instinct scales past one vendor’s tiers, where the open-weight floor handles the cheap turns and frontier keeps the hard ones.

That is a different decision from keeping Claude Code as the harness while choosing GPT at launch, which changes the backend for a whole session rather than routing each call.

So the discipline, restated: pick the cheapest model that clears each subtask’s bar, set that bar with Sonnet before you reach for Haiku, protect the four trap-steps with validation gates instead of optimism, and make every route observable and overridable. The floor on your savings is whatever top-tier step you genuinely can’t downshift, and that is fine, because the goal was never a marketing percentage. It was paying for the intelligence each call actually needs. Wire routing into the per-call meter and the team-wide token budget so you can see it working, because routing without measurement is just a new way to be wrong faster. The same instinct surfaces in the tooling, where local-versus-cloud routing in agent command centers is still a manual call the human makes.