Your Model Benchmarks at 90 Tokens a Second. So Why Does the Chat Window Just Sit There for Two Seconds?

What is time to first token, and why is it different from throughput?

Time to first token (TTFT) is the delay between a user sending a message and the very first token of the model's reply appearing on screen. Throughput, usually reported as tokens per second, measures how fast the model generates text once it has already started, the decode phase. Redis's technical breakdown of prefill versus decode is precise about why the two numbers tell different stories: prefill, the step where the model reads and processes the entire prompt and context before generating anything, is compute-bound and highly parallel, while decode, generating each subsequent token, is memory-bound and strictly sequential because every new token depends on every token that came before it. A model can have excellent decode throughput and still feel sluggish if prefill is slow, because the user is staring at nothing while prefill runs, regardless of how fast decode is afterward.

That prefill step is why context length matters so much for perceived speed: the model has to process every token of the system prompt, conversation history, and any retrieved documents before it can emit token one, so a bloated system prompt or an over-stuffed RAG context directly taxes the metric users actually notice, even if the eventual answer streams out quickly once it starts.

How much do real TTFT numbers actually vary across providers in 2026?

By a lot, and the gap is what actually determines whether an interface feels conversational. Digital Applied's 2026 latency benchmark run put Groq's Llama 4 405B at a 0.18-second median TTFT and Cerebras's Qwen 3 235B at 0.21 seconds, both running on inference hardware purpose-built for speed, against standard hosted endpoints landing much further out: GPT-5.5 standard at 1.12 seconds and Gemini 3 Pro's default endpoint at 0.93 seconds. A model choice that looks identical on a feature-comparison chart can put a full second of dead air in front of every single response a user sends.

The practical implication for anyone building a chat interface or voice agent is that model selection is a latency decision as much as a capability decision, and our breakdown of choosing between GPT, Claude and open-source models for a business agent has to be read alongside whatever TTFT the specific deployment tier actually delivers, not the vendor's average across all their infrastructure.

Why does the P95 latency number matter more than the average?

• A P50, or median, TTFT is the number every vendor puts on the homepage; the same 2026 benchmark run found P95 latency, describing the slowest 1-in-20 requests, inflates 1.6 to 3.2 times over P50 across providers, averaging roughly 2.1 times.
• A user doesn't experience the average of their interactions with a product, they experience each one individually, so a chatbot with a snappy 0.3-second median but a 1-second P95 will still generate a steady trickle of complaints about feeling laggy from the roughly 1-in-20 requests that hit the tail.
• Cheaper, less-optimized inference providers tend to have noisier tails even when their median looks competitive, so a production SLO built only around the median number will look fine in dashboards while a meaningful slice of real users get the slow version.
• For latency-sensitive use cases, voice agents especially, designing around the P95 case rather than the marketing P50, the same discipline covered in our piece on voice agent latency and barge-in architecture, is what prevents the tail from defining the product's reputation.

What can actually be done to cut time to first token in production?

• Trim the system prompt and injected context to the minimum the task needs, since every token in the prompt has to be processed during prefill before the model emits anything.
• Use prompt caching for the static parts of a prompt, system instructions, tool definitions, a large reused document, so the provider skips reprocessing the unchanged prefix on every turn.
• Stream the response over server-sent events so the interface renders tokens as they arrive instead of waiting for the full completion, which won't change true TTFT but changes how fast the delay feels to a user watching the screen.
• Route latency-sensitive turns to a faster, smaller model and reserve the larger reasoning model for turns that genuinely need the extra depth, the same routing logic covered in how LLM model routing cuts AI agent costs, since the fastest available token is often free.

How AIBOOTSTRAPPER helps

Latency isn't an abstraction for us, it's a build requirement. When we shipped AudioBolo end to end, architecture, AI integration, and production launch, cutting average audio processing time from 3.2 seconds to 0.9 seconds was one of the concrete engineering targets, because a product where the user waits and stares doesn't get used a second time regardless of how good the underlying model is. The same discipline, measure the real user-facing delay, not the vendor's benchmark slide, applies directly to text and voice agents.

If your AI agent's model benchmarks well on paper but feels slow in the hands of real users, book a call and we'll help you find out whether it's a prefill problem, a context bloat problem, or the wrong model tier for the job.

Sources and further reading

1. 1.Digital Applied — AI Model Latency Benchmarks 2026: TTFT and Throughput
2. 2.Redis — Prefill vs Decode: LLM Inference Phases Explained