[deleted]
Show HN: I built a sub-500ms latency voice agent from scratch
91–100 of 168 posts
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#92I suprisely noticed that the GitHub repository's name is actually a madarian character 说(speak).
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#93Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…
I've experimented with having different sized LLMs cooperating. The smaller LLM starts a response while the larger LLM is starting. It's fed the initial response so it can continue it. The idea of having an LLM follow and continuously predict the speaker. It would allow a response to be continually generated. If the prediction is correct, the response can be started with zero latency.
Google seems to be experimenting with this with their AI Mode. They used to be more likely to send 10 blue links in response to complex queries, but now they may instead start you off with slop.
(Meanwhile at OpenAI: testing out the free ChatGPT, it feels like they prompted GPT 3.5 to write at length based on the last one or maybe two prompts)
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#94Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…
Regarding 2, I believe that talking on mobile phones drives older people crazy. They remember talking on normal land lines when there was almost no latency at all. The thing is -- they don't know why they don't like it.
Yeah, I remember the time when we had to use satellites to connect. The long delay was really annoying and so unusual that most people without "training" could not even use the phone for conversation and just wasted the dollars.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#95Yea, Deepgram Flux is the secret sauce. Doesn't get talked about much.
For anyone curious: https://flux.deepgram.com/
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#96This is a really solid writeup. The streaming pipeline architecture, the detailed latency breakdown per stage are genuinely useful. Building the core turn-taking loop from scratch is such a good exercise, and you did an excellent job explaining why each part matters and where the actual bottlenecks live. Strongly recommend this to anyone who wants to understand what’s really going on under the hood of a voice agent.
The one spot where it feels a bit off is the "2x faster than Vapi" claim. Your system is a clean straight pipe: transcript -> LLM -> TTS -> audio. No tool calls, no function execution, no webhooks, no mid-turn branching.
Production platforms like Vapi are doing way more work on every single turn. The LLM might decide to call a tool—search a knowledge base, hit an API, check a calendar—which means pausing token streaming, executing the tool, injecting the result back into context, re-prompting the LLM, and only then resuming the stream to TTS. That loop can happen multiple times in a single turn. Then layer on call recording, webhook delivery, transcript logging, multi-tenant routing, and all the reliability machinery you need for thousands of concurrent calls… and you’re comparing two pretty different workloads.
The core value of the post is that deep dive into the orchestration loop you built yourself. If it had just been "here’s what I learned rolling my own from scratch," it would’ve been an unqualified win. The 2x comparison just needs a quick footnote acknowledging that the two systems aren’t actually doing the same amount of work per turn.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#97This is an outstanding write up, thank you! Regarding LLM latency, OpenAI introduced web sockets in their Responses client recently so it should be a bit faster. An alternative is to have a super small LLM running locally on your device. I built my own pipeline fully local and it was sub second RTT, with no streaming nor optimisations https://github.com/acatovic/ova
Very cool! starred and on my reading list. Would love to chat and share notes, if you'd like
You may be interested in gemini-2.5-flash-preview-tts
Text in, audio out, so you can merge in a single step LLM+TTS (streamable)
https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flas...
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#98IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy , but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.
Some of the best current voice tokenizers achieve ~12 Hz, that's many more tokens than a regular LLM would use for ultimately the same content.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#99Yea, Deepgram Flux is the secret sauce. Doesn't get talked about much. For anyone curious: https://flux.deepgram.com/
What is the difference between Flux’s end-of-turn detection and Openai's Automatic turn detection Semantic mode?
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#100The voice samples sound fantastic The interruption handling is amazing. I felt you were talking to an actual person. It might have helped that he had a British accent :)
> The voice samples sound fantastic [...] I felt you were talking to an actual person.
I like to listen to space content when going to sleep. Channels like History of the Universe, Astrum, PBS space time, SEA, etc.
Lately there's been a bunch of new-ish channels that produce content in that space (heh) and I'm amazed of how good the voices sound. Sometimes it takes a few good minutes to figure out they're genai voices, they're that good. If it weren't for small mistakes I bet more than 80% of the general population wouldn't have a clue.