A founder builds an expense-processing agent and, because it's the best model available, routes every single receipt through a frontier-tier model: extract the vendor, the amount, the category, done. The bill at the end of the month is uncomfortable for what the task actually is. Someone finally looks at a sample of the calls and finds that 90% of them are, structurally, "does this line item say Uber or does it say Zomato": a narrow, repetitive classification task, not a task that ever needed a model capable of drafting legal contracts or debugging code. The agent was never wrong. It was just wildly over-equipped for almost everything it was actually being asked to do.
What actually makes a language model "small"?
It isn't a marketing label, it refers to parameter count, roughly under 10 billion parameters, and typically a model trained or fine-tuned toward a narrower task distribution rather than general-purpose fluency across every domain at once. A large generalist model carries enormous, mostly-unused capacity on any single call: it spends the same architecture answering "what's the capital of France" as it does drafting a nuanced legal clause, because it can't know in advance which capability a given request will need. A small model fitted to one narrow, recurring task doesn't carry that unused capacity, and doesn't pay for it in latency or cost either.
Why NVIDIA argues most agent calls don't need a generalist model
NVIDIA's research paper "Small Language Models are the Future of Agentic AI" makes a specific, mechanistic argument, not just a cost observation: agentic systems typically perform specialized tasks repetitively with minimal variation, unlike open-ended, general-purpose LLM applications. An agent isn't a chatbot holding a free-ranging conversation on every call, it's usually the same handful of sub-tasks, classify, extract, format, route, executed thousands of times with small variation between them. That's exactly the profile where a smaller, more specialized model can match accuracy while using a fraction of the compute a generalist model spends per token.
The paper's own recommendation isn't "replace every large model," it's heterogeneous: keep a large model for the genuinely open-ended, conversational parts of a system, and route the narrow, repetitive, high-volume parts to something smaller and cheaper purpose-fit for that task.
What the cost difference actually looks like
Anthropic's own pricing makes the gap concrete without needing a third-party benchmark: Claude Haiku 4.5 is priced at $1 per million input tokens and $5 per million output tokens, versus $5 and $25 for Claude Opus 5, a straight 5x difference on both sides for the same volume of text. Anthropic's own worked example puts this in business terms directly: processing roughly 10,000 support conversations at about 3,700 tokens each on Haiku 4.5 comes out to roughly $37 total. Running that same volume through the frontier-tier model instead of the smaller one doesn't buy more correct answers on a task that's already narrow and repetitive, it just multiplies the bill by five.
A framework: which of your agent's calls are actually candidates for a smaller model?
| Signal | Route to a small/specialized model | Route to a large generalist model |
|---|---|---|
| High call volume, the same task shape every time (classify, extract, tag) | Yes | No |
| Output constrained to a small, fixed set of labels or a strict schema | Yes | No |
| Open-ended writing, nuanced judgment, or multi-step planning | No | Yes |
| Ambiguous input where the model has to weigh several plausible interpretations | No | Yes |
| Cost or latency is the binding constraint at high volume (triage, routing, tagging) | Yes, even if it needs light fine-tuning first | No |
Framework derived from NVIDIA's agentic-AI task-shape argument and Anthropic's published per-model pricing; the right cutoff is still task-specific.
How this shows up in a real build: Expensorr
Expensorr, the AI-powered expense management product AIBOOTSTRAPPER built end to end, is a good example of the exact task shape NVIDIA's paper describes: extracting a vendor, amount, and category from a receipt is a narrow, repeatable classification task, not an open-ended reasoning problem, run at high volume across every user. Building the product to right-size the model behind each step, rather than defaulting to the biggest available model everywhere, is part of what let it go from concept to a production launch in five weeks while saving users over 12 hours a month of manual expense tracking. Full results are on the case studies page.
How AIBOOTSTRAPPER helps
AIBOOTSTRAPPER's AI product development team designs a model portfolio for each build, not a single model applied everywhere, so a client pays frontier prices only for the calls that genuinely need frontier capability.
If your AI agent's bill is climbing faster than the value it's delivering, book a call and we'll look at which of its calls are actually narrow enough to move to a smaller model.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.