A founder demos their new AI voice agent on a quiet office WiFi call, it sounds crisp, responsive, almost human, so they ship it. Two weeks later a support ticket says a customer hung up mid-sentence because the agent kept talking over them, and another says the agent went silent for what 'felt like forever' before answering a simple question. Nothing about the model changed between the demo and production. What changed is that the demo never tested the two things that actually make a voice agent feel human: how fast the full round trip is, and whether it knows how to shut up the instant a caller starts talking.
The latency budget: where every millisecond actually goes
A voice agent's response time isn't one number, it's a stack of four sequential costs: speech-to-text, LLM inference, text-to-speech, and network round trips between whichever vendors handle each step. Telnyx's 2026 benchmarking of voice AI pipelines breaks a typical stitched pipeline down to 100-300ms for STT, 350-1,000ms for LLM inference, 90-200ms for TTS, and another 50-200ms of network overhead between vendors, a total range of roughly 600ms to 1.7 seconds before the caller hears a single word back.
The human conversational baseline is around 200 milliseconds, the natural gap between one person finishing a sentence and the other starting to respond. Telnyx reports that once response time crosses 300-500ms, conversations start to feel unnatural, and above roughly 1.5 seconds, most callers assume the line dropped. A stitched pipeline sitting at 1-1.7 seconds isn't broken in any single component, it's broken by accumulation, four individually reasonable delays that add up to something a caller can feel.
Stitched vs co-located: why the same components perform differently
| Architecture | Typical round-trip | Why |
|---|---|---|
| Stitched (separate STT, LLM, TTS vendors) | 600ms – 1.7s | Each hop adds its own network round trip on top of processing time; audio crosses vendor boundaries multiple times per turn |
| Co-located (single network, regional infra) | Under 200ms achievable on the telephony leg | STT, LLM and TTS stay on the same private network, cutting cross-vendor handoffs and round trips |
| Human conversational baseline | ~200ms | The natural turn-taking gap listeners are calibrated to expect |
Directional latency ranges per architecture. Source: Telnyx, 'Voice AI Agents Compared on Latency (2026 Benchmarks)'.
The three fixes that actually move the number
- Stream every stage instead of waiting for completion: start TTS on the first sentence of the model's response instead of the full completion, and forward STT partial transcripts to the LLM as they finalize rather than waiting for total silence, this is the single biggest lever, since it overlaps stages instead of running them fully in sequence.
- Reduce cross-vendor hops: keeping STT, the LLM call and TTS on infrastructure that's regionally co-located, or at minimum minimizing the number of network boundaries audio has to cross per turn, is what separates sub-200ms telephony legs from multi-hundred-millisecond ones in Telnyx's benchmarking.
- Mask unavoidable delay instead of leaving dead air: when a function call (a calendar lookup, a CRM query) genuinely takes a second or two, a queued filler response ('let me check that') covers the gap, dead air reads to a caller as a dropped call far faster than it reads as 'thinking.'
- Pick a fast model for the conversational turn, not the smartest one: a model with a 1000ms+ time-to-first-token adds more perceived delay than most callers will tolerate, even if it produces a marginally better answer, latency-optimized model variants exist specifically for this reason.
How barge-in actually works
Barge-in is the agent's ability to stop talking the instant a caller starts speaking, the same reflex a human has mid-conversation. Structurally, it requires the system to keep listening to the inbound audio stream continuously, even while the agent's own TTS output is playing, and to interrupt that output the moment new speech is detected, discarding whatever was queued to say next and starting a fresh transcription pass on the caller's new input.
FutureAGI's 2026 implementation guide to barge-in and turn-taking sets the production bar at a 200-400ms turn-taking gap (how quickly the agent yields the floor once interrupted), a false-barge-in rate under 2% (background noise or a caller's 'mm-hmm' incorrectly triggering an interrupt), and a TTS flush time under 60ms, how fast the agent's own audio actually stops playing once an interruption is detected. Miss any of the three and the agent either talks over callers, cuts itself off on every stray noise, or has an audible lag between 'caller starts talking' and 'agent actually stops.'
Where barge-in breaks in practice
- False positives from background noise: a caller in a car or a noisy office triggers constant false interruptions if the system isn't using proper voice activity detection tuned against non-speech audio, not just any incoming sound.
- Filler words treated as real interruptions: a caller saying 'mm-hmm' or 'right' while the agent is still talking shouldn't cut the agent off, distinguishing backchannel acknowledgment from an actual attempt to speak requires more than a raw volume threshold.
- TTS that doesn't actually stop: some implementations detect the interruption correctly but keep the queued audio buffer playing for another second because the flush logic isn't wired to the same real-time path as the detection, the caller experiences this as the agent 'not listening' even though the system technically noticed.
- No graceful re-entry: after a successful barge-in, the agent needs to process the caller's new input as the priority, not awkwardly try to finish its original sentence once the caller stops talking again.
How AIBOOTSTRAPPER helps
This is the gap between a voice agent that works in a demo and one that survives real call volume, real background noise and real interruptions, tuned endpointing, genuine end-to-end streaming, and barge-in that behaves the way a caller expects without a wall of custom infrastructure work most teams don't budget for. Our step-by-step guide to building an inbound voice agent covers the base architecture; getting the latency budget and interruption handling production-ready is the layer on top of that most teams underestimate until callers start complaining.
AIBOOTSTRAPPER's AI automation team builds and tunes voice agents against your actual call volume and script, not a generic template. Book a call if your current voice agent sounds fine in a demo but is losing callers in production.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.