Show HN: I built a sub-500ms latency voice agent from scratch

#1

Post by nicktikhonov » Mon, Mar 02, 2026, 9:23 PM UTC

I built a voice agent from scratch that averages ~400ms end-to-end latency (phone stop → first syllable). That’s with full STT → LLM → TTS in the loop, clean barge-ins, and no precomputed responses.

What moved the needle:

Voice is a turn-taking problem, not a transcription problem. VAD alone fails; you need semantic end-of-turn detection.

The system reduces to one loop: speaking vs listening. The two transitions - cancel instantly on barge-in, respond instantly on end-of-turn - define the experience.

STT → LLM → TTS must stream. Sequential pipelines are dead on arrival for natural conversation.

TTFT dominates everything. In voice, the first token is the critical path. Groq’s ~80ms TTFT was the single biggest win.

Geography matters more than prompts. Colocate everything or you lose before you start.

GitHub Repo: https://github.com/NickTikhonov/shuo

Follow whatever I next tinker with: https://x.com/nick_tikhonov

Show HN: I built a sub-500ms latency voice agent from scratch
ntik.me

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#2

Post by MbBrainz » Mon, Mar 02, 2026, 9:31 PM UTC

Love it! Solving the latency problem is essential to making voice ai usable and comfortable. Your point on VAD is interesting - hadn't thought about that.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#3

Post by NickNaraghi » Mon, Mar 02, 2026, 9:34 PM UTC

Pretty exciting breakthrough. This actually mirrors the early days of game engine netcode evolution. Since latency is an orchestration problem (not a model problem) you can beat general-purpose frameworks by co-locating and pipelining aggressively.

Carmack's 2013 "Latency Mitigation Strategies" paper[0] made the same point for VR too: every millisecond hides in a different stage of the pipeline, and you only find them by tracing the full path yourself. Great find with the warm TTS websocket pool saving ~300ms, perfect example of this.

[0]: https://danluu.com/latency-mitigation/

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#4

Post by jangletown » Mon, Mar 02, 2026, 9:52 PM UTC

impressive

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#5

Post by lukax » Mon, Mar 02, 2026, 10:25 PM UTC

Or you could use Soniox Real-time (supports 60 languages) which natively supports endpoint detection - the model is trained to figure out when a user's turn ended. This always works better than VAD.

https://soniox.com/docs/stt/rt/endpoint-detection

Soniox also wins the independent benchmarks done by Daily, the company behind Pipecat.

https://www.daily.co/blog/benchmarking-stt-for-voice-agents/

You can try a demo on the home page:

https://soniox.com/

Disclaimer: I used to work for Soniox

Edit: I commented too soon. I only saw VAD and immediately thought of Soniox which was the first service to implement real time endpoint detection last year.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#6

Post by nicktikhonov » Mon, Mar 02, 2026, 10:29 PM UTC

lukax wrote: post #5

Or you could use Soniox Real-time (supports 60 languages) which natively supports endpoint detection - the model is trained to figure out when a user's turn ended. This always works better than VAD. https://soniox.com/docs/stt/rt/endpoint-detection Soniox also wins the independent benchmarks done by Daily, the company behind Pipecat. https://www.daily.co/blog/benchmarking-stt-for-voice-agents/ You can try a demo on t…

If you read the post, you'll see that I used Deepgram's Flux. It also does endpointing and is a higher-level abstraction than VAD.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#7

Post by loevborg » Mon, Mar 02, 2026, 10:35 PM UTC

Nice write-up, thanks for sharing. How does your hand-vibed python program compare to frameworks like pipecat or livekit agents? Both are also written in python.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#8

Post by lukax » Mon, Mar 02, 2026, 10:37 PM UTC

nicktikhonov wrote: post #6

lukax wrote: post #5

Or you could use Soniox Real-time (supports 60 languages) which natively supports endpoint detection - the model is trained to figure out when a user's turn ended. This always works better than VAD. https://soniox.com/docs/stt/rt/endpoint-detection Soniox also wins the independent benchmarks done by Daily, the company behind Pipecat. https://www.daily.co/blog/benchmarking-stt-for-voice-agents/ You can try a demo on t…

If you read the post, you'll see that I used Deepgram's Flux. It also does endpointing and is a higher-level abstraction than VAD.

Sorry, I commented too soon. Did you also try Soniox? Why did you decide to use Deepgram's Flux (English only)?

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#9

Post by perelin » Mon, Mar 02, 2026, 10:40 PM UTC

Great writeup! For VAD did you use heaphone/mic combo, or an open mic? If open, how did you deal with the agent interupting itself?

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#10

Post by nicktikhonov » Mon, Mar 02, 2026, 10:41 PM UTC

lukax wrote: post #8

nicktikhonov wrote: post #6

Earlier quoted context omitted.

If you read the post, you'll see that I used Deepgram's Flux. It also does endpointing and is a higher-level abstraction than VAD.

Sorry, I commented too soon. Did you also try Soniox? Why did you decide to use Deepgram's Flux (English only)?

I didn't try Soniox, but I made a note to check it out! I chose Flux because I was already using Deepgram for STT and just happened to discover it when I was doing research. It would definitely be a good follow-up to try out all the different endpointing solutions to see what would shave off additional latency and feel most natural.

Another good follow-up would be to try PersonaPlex, Nvidia's new model that would completely replace this architecture with a single model that does everything:

https://research.nvidia.com/labs/adlr/personaplex/