Model Routing Is Simple. Until It Isn’t.

What Happened and Why It Matters

IBM Research published a detailed postmortem on building model routers for agentic systems — and the findings upend a lot of conventional wisdom. What looks like a straightforward classification problem — send simple requests to cheap models, complex ones to expensive models — turns out to be a multidimensional systems optimization challenge. The team ran experiments on the AppWorld Test Challenge using a CodeAct agent across 417 tasks, comparing Claude Sonnet 4.6 and GPT-4.1, and discovered that sticker-price assumptions are dangerously misleading. Sonnet ended up costing $0.19 per task versus GPT-4.1’s $0.37 per task, despite having higher listed token prices and taking roughly three times as many reasoning steps.

Why does this matter to developers who build with AI? Because routing is becoming a central architectural decision as teams deploy multiple models in production. If you’re building agentic systems — tools that chain reasoning, tool use, and retrieval — you’re likely already considering or experimenting with routing. The IBM Research findings show that the common heuristics (price per token, model size, task difficulty) are insufficient. The cost, latency, and quality you actually observe depend on interplay between workload patterns, caching infrastructure, governance rules, and routing granularity. Getting routing right can save significant operational expense; getting it wrong can double costs without improving quality.

The Three Hidden Dimensions of Model Routing

1. Real cost depends on caching, not token pricing. The IBM team’s headline result is counterintuitive: Sonnet cost less than GPT-4.1 despite being slower and more expensive on paper. The culprit is token caching. Agent workloads reuse large blocks of context across steps, so cache hit rates dominate effective input costs. Sonnet’s lower cache-read pricing turned its longer trajectories into an advantage — every reasoning step that reused cached context cost much less. GPT-4.1’s lower base price didn’t help because its cache pricing couldn’t offset the pattern. The implication is clear: a router that optimizes only against published pricing sheets is flying blind. You need to measure the actual cost-per-task under your specific workload and serving setup, including cache behavior.

2. Task complexity is invisible at routing time — and multi-objective. The instinct to route by estimated difficulty sounds logical but breaks down in practice. A simple-sounding request like “summarize this contract” can explode into a multi-step pipeline involving retrieval, compliance checks, tool calls, and iterative refinement. Conversely, a technically dense query might be perfectly handled by a smaller specialized model. Difficulty often only reveals itself during execution. Even if you could predict it perfectly, difficulty is just one signal among many. Production routers also have to balance latency budgets, cost constraints, model specialization, compliance rules, data residency, and approved-model lists. A task that would ideally go to a powerful general model might need to be rerouted because of privacy or governance requirements. The router isn’t solving a single optimization — it’s constantly juggling conflicting priorities.

3. Latency is about infrastructure, not just model size. Conventional thinking says bigger models are slower. But end-to-end latency is dominated by routing overhead, hardware availability, cache warmth, and endpoint load. A theoretically faster model can deliver a worse user experience if its serving conditions are suboptimal. Moreover, routing granularity adds its own complexity. Routing once per task is lightweight, but routing at every step gives more flexibility to adapt — at the cost of additional latency and operational complexity with each decision. A router that ignores the serving system is optimizing against the wrong reality.

What Developers Should Do Now

These findings point to a concrete set of actions for anyone building — or planning to build — a model router in an agentic system.

• Instrument your workload. Don’t rely on model card pricing. Measure actual cost-per-task, including cache hit rates, input/output token distributions, and the number of reasoning steps required. These metrics vary dramatically across agents and domains.
• Don’t model routing as a classification problem alone. Treat it as a multi-objective optimization problem that adapts based on real-time signals: latency, cost, quality scores, and compliance constraints. That may mean building a simple reinforcement learning loop or using a meta-model that can weigh tradeoffs dynamically.
• Account for caching in your cost model. If your agent reuses context heavily — which many do, especially with system prompts and tool definitions — the cache pricing of different providers can flip the economic picture. Consider caching behavior as part of model selection, not an afterthought.
• Design routing granularity carefully. Per-task routing is simpler but less responsive. Per-step routing offers more control but adds latency and complexity. Measure the overhead of routing at your target granularity before committing.

For a deeper comparison of how different coding agents handle routing and tool use, see our analysis of the best AI coding agents in 2026. That piece evaluates how agents like CodeAct, Claude Code, and others manage multiple models and tool-calling orchestration in practice.

The IBM Research team’s overall message is that routing should be approached experimentally. They recommend implementing a simple proof-of-concept router first, instrumenting it heavily, and then iterating. Empirically measuring costs, latency, and quality under representative conditions is far more reliable than theorizing about difficulty or pricing.

The Bigger Picture

Model routing is a linchpin in the emerging architecture of multi-model agents. As the ecosystem matures, we’re seeing more tools that specialize by domain — some models excel at code generation, others at multimodal understanding, and still others at structured reasoning. The promise of routing is to get the best of all worlds without paying for the most expensive model on every task. But the IBM Research post demonstrates that this promise is harder to deliver than many expect.

If you’re considering using a tool like Cursor or Claude Code for agentic workflows, the routing logic built into those tools matters. The way they delegate subtasks to different models — or choose between models for code editing versus explanation — can have a big impact on both cost and quality. For a head-to-head comparison of two popular setups, check out Cursor vs Claude Code to see how their routing strategies differ and what that means for your team.

The takeaway for builders is not to give up on routing, but to approach it with humility and measurement. Start small. Measure everything. Expect your assumptions about model costs and latency to be wrong. And treat routing as a systems problem that evolves with your workload, your serving stack, and your compliance requirements. The teams that will win with multi-model agents are the ones that treat routing as a continuous optimization experiment — not a one-time classification fix.

Source: Hugging Face. Details as reported; verify specifics at the source.