An engineer wires up a support agent with two tools, refund_order and cancel_order, writes a system prompt that spells out exactly when to use each one, tests it on ten cases, ships it. Week two, a customer asks for a refund on a damaged item and the agent cancels the order instead, no refund issued, no error thrown, just a wrong action executed with total confidence. The engineer rereads the system prompt, rewrites it to be even more explicit, redeploys, and the same failure shows up on a different case a few days later. The prompt was never the layer where this decision got made, and that's the part almost nobody checks first.
Why rewriting the system prompt doesn't fix it
Tool selection in a function-calling agent is not decided by the system prompt, it's decided by the model reading the tool's name, description and parameter schema at the moment it chooses which function to call, and weighing that against the conversation so far. Anthropic's own tool use documentation frames tool definitions as the primary interface the model reasons over, the system prompt sets general behavior, but the tool schema is what actually gets compared against the user's request during selection.
That's why the standard debugging instinct, edit the system prompt, targets the wrong layer. A recent mechanistic study of tool-selection failures, 'Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents', found something specific: when researchers measured which tool the model was actually attending to at the moment of failure, it was attending to the correct tool roughly 80% of the time, and still picked the wrong one. The model looked at the right tool and chose differently anyway, which means the failure sits at the decision step, not at retrieval or attention, and no amount of system-prompt rewriting touches that step.
The four failure modes, and which one you actually have
- Incorrect tool selection: the right tool exists and is visible to the model, but a similar-sounding or overlapping tool gets called instead, refund_order vs cancel_order, search_customer vs get_customer. This is the failure the arXiv study above is describing, and it's a description problem, not a prompt problem.
- Bad arguments: the correct tool gets called, but with malformed or wrong-typed arguments, a date string where an ISO timestamp was expected, a customer name where an ID was required. This gets worse as schemas get more complex or the model is smaller, and it compounds silently unless arguments are validated before execution.
- Missing context: the agent calls a tool with a stale or incomplete picture of the conversation, using an order ID from three turns ago instead of the one the customer just mentioned, because nothing in the flow forced it to re-anchor before acting.
- Too many tools: past roughly 20-30 tools exposed to the model in a single call, both token cost and wrong-tool-selection rate climb, because the model is now discriminating between a much larger and often more overlapping set of options at once. This is a real enough problem that Anthropic shipped a dedicated Tool Search feature that lazy-loads a relevant subset of tools instead of exposing the full set on every call.
What actually fixes tool selection
Since the failure lives in the tool definition, not the system prompt, the fixes live there too. None of these require a bigger model, they require a more disciplined schema.
| Fix | What it does | Why it works |
|---|---|---|
| Rewrite descriptions as 'when to use', not 'what it does' | "Cancels an order before it ships and issues no refund" instead of "Cancels an order" | Gives the model the discriminating condition it needs at the decision step, not just a label |
| Rename ambiguous parameters | query becomes search_term, id becomes customer_id | Parameter names carry semantic weight on their own; a generic name invites a broader, wronger interpretation |
| Cap and group the tool set | Route to a relevant subset (10-15 tools) instead of exposing everything on every call | Keeps the model discriminating between a small, low-overlap set instead of 20-30+ similar options |
| Validate arguments before execution | A JSON Schema check on every tool call's arguments before it runs | Catches type mismatches and malformed calls pre-execution instead of letting them fail downstream, silently or loudly |
| Add a response-check step | Verify the tool's actual output matches the resource that was asked for | Catches the case where the right tool ran but on the wrong record, an error class validation alone won't catch |
Fixes mapped to the failure mode they actually address, ordered roughly by implementation effort.
Why this matters more once agents take real actions
A wrong tool call that returns a wrong text answer is embarrassing. A wrong tool call that cancels an order, sends a refund, or fires off a message is a production incident, and it happens with the exact same model confidence as a correct call, there's no hesitation signal to catch it on the way out. That's the core argument for evaluating an agent with trace-based testing before it ships rather than trusting that ten manually-checked demo cases represent the long tail of real requests, and for building structured output validation as a standing layer rather than a one-time check.
This is also why multi-step, function-calling agent architecture needs a harder line between what an agent can decide autonomously and what it must confirm before executing, irreversible actions (refunds, cancellations, outbound messages, data deletion) deserve either a confirmation step or a narrower, purpose-built tool that structurally can't be confused with a neighboring one.
How AIBOOTSTRAPPER solved this for a client
AI Doctor, a telehealth platform we built for a Dubai clinic, runs on exactly this kind of high-stakes tool selection: the agent has to choose correctly between triage actions like 'suggest self-care', 'schedule a routine consult' and 'escalate to a human doctor immediately', where picking the neighboring tool instead of the right one isn't a minor bug, it's a patient safety issue. We built the assistant with clinically guarded tool definitions, narrow, non-overlapping actions with explicit 'when to use' descriptions, and a mandatory human handoff path with a pre-filled summary rather than letting the agent silently resolve ambiguous cases on its own. That discipline is a direct reason the platform cut consultation prep time by 68% while keeping a 24/7 triage layer that doctors actually trust.
If your agent's tool calls are silently going wrong, wrong refund, wrong record updated, wrong escalation, AIBOOTSTRAPPER's AI product development team audits your actual tool schemas and failure logs, not just your prompts, and rebuilds the tool layer so the model has a real discriminating signal to work with. Book a call to get your agent's tool-calling reliability audited.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.