A team ships an AI agent that handles both "what's my order status" (should be instant) and "should we approve this refund exception given three conflicting policy clauses" (should be careful) through the exact same model call, because building one path was simpler than building two. The refund agent is fast and confidently wrong close to half the time; the status-check agent is accurate but takes several seconds and costs several times what a lookup should. Both problems trace back to the same missing decision: nobody decided, on purpose, which calls should actually be allowed to think before answering.
What does a reasoning model actually do differently?
A reasoning-enabled model generates internal reasoning tokens before its final answer, working through the problem, weighing approaches, checking itself, and those tokens are billed as output but mostly hidden from the end user. Per Anthropic's documentation on extended thinking, you set a token budget for this internal reasoning (1,024 tokens minimum) that Claude works against before producing its visible response; OpenAI's reasoning guide describes a comparable mechanism where reasoning tokens let the model "break down the prompt and consider multiple approaches to generating a response" before the answer starts.
The model isn't accessing new information by thinking longer, it's spending more compute exploring and checking its own reasoning path before committing to an answer. That's why it helps on multi-step, ambiguous or high-stakes problems, and does close to nothing for a lookup that only has one correct answer to begin with.
Why turning this on for every call is the wrong default
Reasoning tokens are billed as output tokens and add latency roughly proportional to how long the model reasons. OpenAI's own guidance frames effort as a dial: low settings are optimized for speed and lower token usage, medium is described as "a well-balanced point on the pareto curve of latency, performance and cost" and is the default for its newer models, while high and above settings prioritize quality over latency, using "fewer tokens for simpler tasks and think[ing] harder for complex tasks."
Anthropic's tuning guidance is similar in spirit: start near the 1,024-token minimum for simple tasks and increase incrementally, but for genuinely complex tasks start at 16,000 tokens or more, since higher budgets bring diminishing returns at the cost of latency, and thinking budgets above 32,000 tokens should run through batch processing to avoid request timeouts.
A simple framework: when does a call actually need to think?
| Signal | Route to a reasoning/thinking call | Route to a fast, standard call |
|---|---|---|
| Multiple plausible answers that need weighing (policy exceptions, ambiguous approvals) | Yes | No |
| Single correct answer already sitting in a database or API (order status, account balance) | No | Yes |
| Multi-step planning across several tool calls | Yes | No |
| High-volume, low-stakes classification or extraction | No | Yes |
| A wrong answer is expensive (a financial exception, a triage escalation, a legal-adjacent decision) | Yes, even if slower | No |
Framework derived from OpenAI's and Anthropic's own reasoning-effort guidance; the right threshold is still task-specific, not a fixed rule.
The production pattern: route individual steps, don't switch the whole agent to reasoning
Production agents increasingly route each step to the model and effort level suited to it rather than picking one setting for the entire conversation, the same model-routing discipline already used to control cost. A fast, low-effort call handles triage and classification; a reasoning call is reserved for the one step in the flow that actually requires deliberation, the final approval decision, not every turn that led up to it. Choosing which model to reason with at all is the prerequisite decision covered in choosing an AI model for a business agent.
How this discipline shows up in an agent like AI Doctor
AI Doctor, AIBOOTSTRAPPER's bilingual symptom-triage assistant, has to make exactly the kind of multi-factor judgment call worth spending extra reasoning on: whether a given combination of symptoms warrants an urgent human escalation versus a routine note for the doctor. The surrounding conversational turns, collecting basic patient details, don't need that same depth. Treating the escalation judgment as the one step worth the extra latency, rather than applying it uniformly across the whole conversation, is part of what lets the handoff workflow cut consultation prep time by 68% without slowing down the routine parts of the interaction.
How AIBOOTSTRAPPER helps
AIBOOTSTRAPPER's AI product development team designs agent architectures where reasoning is a deliberate routing decision per step, not a blanket setting, so a client's agent is fast where speed matters and careful where a mistake actually costs something.
If your AI agent feels either too slow everywhere or confidently wrong on the calls that matter most, book a call and we'll look at where reasoning should and shouldn't be turned on.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.