Voice agents: cascaded vs speech-to-speech
In conversation, people usually start replying within a few hundred milliseconds of the other person stopping, often around 200 milliseconds. A 2009 study of ten languages, from Japanese to Danish, found the same pattern in all of them: overlaps avoided, silences kept short, and average gaps within about a quarter of a second of each other. A text chatbot that takes two seconds to answer feels fine. A voice agent that takes two seconds feels like a bad phone line: the caller says "hello?", talks over it, and the agent answers a question nobody is asking any more.
That makes voice the hardest latency problem in AI engineering. There are two ways to build it. A cascaded pipeline chains three models: speech-to-text (STT) turns audio into words, an LLM writes the reply, and text-to-speech (TTS) speaks it. A speech-to-speech model takes audio in and produces audio out directly, with no text stage in between. Most production voice agents today are cascaded, and knowing why, and when that is changing, is what this page is about.
A bank's voice agent takes 2.4 seconds from the caller finishing a sentence to the first word of the reply. Callers keep interrupting it. Without measuring yet, which part of the pipeline would you suspect first, and why?
Where the second goes#
A cascaded turn has more stages than three models, and the biggest delay is usually not a model at all. It is deciding that the user has finished speaking.
Callers will talk over this. The biggest share is the LLM, so that is where to cut first.
The common targets: under about 800 ms voice to voice feels natural, up to around 1.5 seconds is tolerable, and beyond that people start talking over the agent. Every stage has to be streaming, and the stages must overlap: STT transcribes while the caller speaks, the LLM starts as soon as the turn ends, and TTS starts on the first sentence while the LLM is still writing the rest.