A team running an AI agent on GPT-4 or Claude Opus for every single message watches the bill triple as usage grows, and someone proposes the obvious fix: swap in a cheaper model. Two weeks later, support tickets pile up because the cheap model quietly botches the 15% of conversations that actually needed real reasoning, and the team reverts to the expensive model everywhere, more scared of the failure than the cost. The fix that actually works sits between those two extremes: a routing layer that sends each query to the cheapest model capable of handling it, not the same model for every query regardless of difficulty.
What is LLM model routing, mechanically?
A router sits in front of two or more models, typically one frontier model (GPT-4 or Claude Opus class) and one smaller, cheaper model (GPT-4o mini or Claude Haiku class), and for every incoming query it makes one decision before the actual model call: which model should answer this. The query still gets a real answer from a real model, the only thing that changes is which model gets picked, based on how complex the request actually is.
This is distinct from simply picking a cheaper model for your whole product. RouteLLM, an open-source framework from LMSYS, trains this decision directly on preference data: pairs of responses from a strong and a weak model, labeled by which one a human (or another LLM acting as judge) actually preferred, so the router learns to predict, before either model runs, whether the cheap model's answer would already be good enough.
The three ways routers actually decide
- Rule-based routing: simple heuristics on query length, keyword presence, or task type (a one-line FAQ lookup vs a multi-step analysis request), the cheapest to build and run, adding effectively no latency, but blunt, since it can't judge difficulty within a category.
- Embedding-based routing: the query is embedded and compared against reference clusters of known easy and known hard queries, a small extra model call before the real one, more accurate than pure rules at a modest latency cost.
- Preference-trained routers (the RouteLLM approach): a router model trained on thousands of strong-vs-weak response comparisons learns a general scoring function for 'would the cheap model's answer satisfy this specific request', which generalizes to queries the router has never seen before, not just queries matching a hand-written rule.
The actual numbers, and why they matter more than they look
The RouteLLM paper tested routing between GPT-4 and the open-weight Mixtral 8x7B, and its matrix-factorization router, trained purely on public Chatbot Arena preference data, reached 95% of GPT-4's performance on MT-Bench while sending only 26% of queries to GPT-4 itself. Augmenting the training data with LLM-judge labels pushed that down further, to just 14% of queries needing the expensive model for the same quality bar.
On other benchmarks the same routers cut GPT-4 calls by over 85% on MT-Bench, 45% on MMLU and 35% on GSM8K relative to an all-GPT-4 baseline, which is the real headline: cost and quality are not a straight-line tradeoff once you route per-query instead of per-product. The wrong way to read these numbers is 'always use the cheap model'; the right way is 'a well-trained router finds the 14-26% of queries that actually need the expensive model and sends everything else to the cheap one.'
What actually breaks a naive routing setup
- Downgrading instead of escalating: a router that commits to the cheap model's answer with no fallback will occasionally ship a wrong answer with full confidence. Production routers should escalate to the expensive model when the cheap model's confidence or self-consistency score is low, not silently accept whatever the cheap model returns.
- Ignoring routing latency in the total budget: as TrueFoundry's breakdown of cost-and-quality-aware routing notes, rule-based routing adds negligible latency, but an embedding call or a full classifier model adds real milliseconds on every request, which matters for latency-sensitive agents even when it barely moves the cost line.
- Training the router on the wrong distribution: a router trained on generic benchmark preference data (MT-Bench, Arena) may not generalize to a narrow, domain-specific product like legal drafting or medical triage, where 'easy' and 'hard' look nothing like general chat.
- Treating cost per token as the metric instead of cost per successful output: a cheap model that fails a task 30% of the time and needs a retry (at extra cost and latency) can end up more expensive per completed task than a pricier model that succeeds on the first try, even though its per-token price looks lower.
Routing vs prompt caching vs semantic caching: three different levers
These get bundled together but solve different problems. Model routing decides which model answers a query. Prompt caching skips recomputing an unchanged prefix on repeated calls to the same model. Semantic caching skips the model call entirely when a new question is similar enough to one already answered. A cost-aware production system layers all three: route by difficulty, cache the stable system prompt, and cache the actual repeated answers on top, rather than treating any single lever as the whole fix.
How AIBOOTSTRAPPER helps
Routing only pays off when the classifier is tuned to your actual traffic, not a generic benchmark, which means auditing real query logs before picking thresholds. AIBOOTSTRAPPER's AI product development team builds this routing and escalation logic into agent architectures from the first sprint, alongside the caching layers covered in the API cost playbook, rather than retrofitting it after a surprise bill.
If you're running an agent on a single frontier model for every request and want a second opinion on where routing would actually save money without hurting quality, book a call.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.