A team ships an AI agent with a 2,400-token system prompt and a dozen tool definitions, resent in full on every single turn of every conversation. The bill triples in the first month of real usage. Someone reads 'enable prompt caching' in the docs, flips it on, and is confused two days later to see a line item for cache writes costing more than a normal input token, not less. Prompt caching isn't a discount switch, it's a specific mechanism with its own cost structure, and understanding what's actually happening under the hood is the difference between a 90% cost cut and a rounding error.
What is prompt caching, mechanically?
Every request to an LLM has to run the full input through its attention layers to build the internal key/value representations it needs to generate a response, even if 90% of that input is identical to the request sent thirty seconds earlier. Prompt caching marks a stable prefix, typically the system prompt, tool definitions and any fixed instructions, with a boundary, and if a new request's tokens match that prefix exactly, the provider skips recomputing it and reuses the stored key/value state instead.
Anthropic's implementation requires an explicit `cache_control` marker at the point in the prompt where the stable content ends. OpenAI's newer models route requests automatically based on a hash of the first 256 tokens, with an optional `prompt_cache_key` parameter to make matching more reliable for high-traffic agents.
The pricing math nobody explains upfront
| Token type | Relative cost | Example: Claude 3.5 Sonnet |
|---|---|---|
| Normal input token | 1x (baseline) | $3.00 / MTok |
| Cache write (first time) | 1.25x | $3.75 / MTok |
| Cache read (subsequent hits) | 0.10x | $0.30 / MTok |
Cache write is a one-time 25% premium to store the prefix; every matching request after that reads it at a 90% discount, per Anthropic and OpenAI's published pricing structures.
Why cache writes cost more, and why that's still a win
The write premium exists because storing the key/value state for reuse is real infrastructure cost the provider passes through once, not a penalty. The economics work because you pay the 1.25x premium a single time per cache lifetime, then every subsequent request that hits the same prefix pays 0.10x instead of 1x, a 90% reduction on that portion of the input.
The breakeven is fast: with Claude's numbers, one cache write plus a single cache read already costs less than two uncached calls ($3.75 + $0.30 = $4.05 vs $3 + $3 = $6), and every hit after that is close to pure savings. Anthropic reports up to 90% cost reduction and up to 85% latency reduction for long, reused prompts, and workloads with system prompts or tool schemas reused across hundreds of requests a day see that ceiling in practice, not just on the pricing page.
What actually breaks a cache hit
- Any change to a token before the cache boundary invalidates the whole prefix, a timestamp embedded in the system prompt, a dynamically reordered tool list, or even inconsistent JSON key ordering in a serialized schema will silently drop you to a full-price cache write on every request.
- TTL expiration: Anthropic's default cache lifetime is short (minutes, extendable to an hour on request); OpenAI's automatic caching defaults to a 30-minute eligibility window, extendable up to 24 hours for high-traffic prefixes. A prefix that isn't hit again inside that window falls out of cache and the next request pays the write cost again.
- Minimum prefix length: OpenAI requires at least 1,024 tokens before a prefix is even eligible for caching, so a short system prompt with no tool schema often isn't worth caching at all, the savings only show up once your stable prefix is genuinely large.
- Low-traffic prefixes: if a given system prompt variant is only used once every few hours, the cache expires between uses and you're paying the write premium on every call with none of the read discount, which is why per-tenant or per-customer prompt variants often cache worse than a single shared prompt.
Prompt caching vs semantic caching: different mechanism, different problem
These get conflated but solve different things. Prompt caching, covered here, skips recomputation for an exact, byte-identical prefix, it's a token-level infrastructure optimization. Semantic caching, covered in the broader API cost playbook, stores embeddings of past questions and returns a cached answer when a new question is similar enough by cosine similarity, skipping the model call entirely rather than just the prefix. A production system with real cost discipline uses both: prompt caching for the stable system prompt and tool schema on every call, semantic caching for the actual repeated user questions layered on top.
How AIBOOTSTRAPPER solved this for a client
AI Doctor, the Dubai-based clinical triage assistant AIBOOTSTRAPPER built, runs 24/7 with a long, carefully tuned clinical safety system prompt, bilingual instructions and escalation guardrails that stay identical across every patient conversation. That's close to a best-case prompt-caching workload: a large, stable prefix reused on effectively every call, which is a meaningful part of how the product sustains round-the-clock availability without the per-conversation cost scaling linearly with traffic.
How AIBOOTSTRAPPER helps
AIBOOTSTRAPPER's AI product development team architects caching, both prompt-level and semantic, into an agent's cost structure from the first build, not as a retrofit after the first surprise bill.
If you already have an agent in production and aren't sure whether your cache hit rate is actually working, book a call and we'll audit it against the token-level breakdown above.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.