Show HN: I built a sub-500ms latency voice agent from scratch • Hacker News Board

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#101

Post by cootsnuck » Tue, Mar 03, 2026, 6:15 AM UTC

modeless wrote: post #14

IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy , but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.

I've been working solely on voice agents for the past couple years (and have worked at one of the frontier voice AI companies).

The cascading model (STT -> LLM -> TTS), is unlikely to go away anytime soon for a whole lot of reasons. A big one is observability. The people paying for voice agents are enterprises. Enterprises care about reliability and liability. The cascading model approach is much more amenable to specialization (rather than raw flexibility / generality) and auditability.

Organizations in regulated industries (e.g. healthcare, finance, education) need to be able to see what a voice agent "heard" before it tries to "act" on transcribed text, and same goes for seeing what LLM output text is going to be "said" before it's actually synthesized and played back.

Speech-to-Speech (end-to-end) models definitely have a place for more "narrative" use cases (think interviewing, conducting surveys / polls, etc.).

But from my experience from working with clients, they are clamoring for systems and orchestration that actually use some good ol' fashioned engineering and that don't solely rely on the latest-and-greatest SoTA ML models.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#102

Post by ghrl » Tue, Mar 03, 2026, 6:30 AM UTC

jedberg wrote: post #68

stavros wrote: post #66

Earlier quoted context omitted.

I'm puzzled by this conversation, because Amazon did get on the agent bandwagon with Alexa Plus (I have it, it's buggier than regular Alexa and it's all making me throw my Echos away since they can't even play Spotify reliably). Also, my Alexa does advertise stuff to me when I talk to it. It's not Budweiser, but it'll try to upsell me on Amazon services all the time.

> because Amazon did get on the agent bandwagon with Alexa Plus Which just launched last year, about four years after ChatGPT had AI voice chat. And it costs extra money to cover the costs. And as you aptly point out, all the guardrails they had to put in made the experience less than ideal. > Also, my Alexa does advertise stuff to me when I talk to it. Yes, that is how they try to make money. And it's gotten worse.…

I would say that depends. When it tries to upsell Prime subscriptions into even more Amazon subscriptions I always interrupt it and say the command again so it stops, but a few times it told me "this item in your cart is on sale by some %" and that did make me buy the item.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#103

Post by azinman2 » Tue, Mar 03, 2026, 6:31 AM UTC

YesBox wrote: post #80

jedberg wrote: post #36

Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…

Why dont voice assistants use a finishing word or sound? People are already trained to say a name to start. Curious why the tech has avoided a cap? “Alexa, what’s tomorrow’s weather [dada]?”

Because that’s extremely unnatural.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#104

Post by cootsnuck » Tue, Mar 03, 2026, 6:34 AM UTC

totetsu wrote: post #99

cootsnuck wrote: post #95

Yea, Deepgram Flux is the secret sauce. Doesn't get talked about much. For anyone curious: https://flux.deepgram.com/

What is the difference between Flux’s end-of-turn detection and Openai's Automatic turn detection Semantic mode?

In OpenAI's own words about semantic_vad:

> Chunks the audio when the model believes based on the words said by the user that they have completed their utterance.

Source: https://developers.openai.com/api/docs/guides/realtime-vad

OpenAI's Semantic mode is looking at the semantic meaning of the transcribed text to make an educated guess about where the user's end of utterance is.

According to Deepgram, Flux's end-of-turn detection is not just a semantic VAD (which inherently is a separate model from the STT model that's doing the transcribing). Deepgram describes Flux as:

> the same model that produces transcripts is also responsible for modeling conversational flow and turn detection.

[...]

> With complete semantic, acoustic, and full-turn context in a fused model, Flux is able to very accurately detect turn ends and avoid the premature interruptions common with traditional approaches.

Source: https://deepgram.com/learn/introducing-flux-conversational-s...

So according to them, end-of-turn detection isn't just based on semantic content of the transcript (which makes sense given the latency), but rather the the characteristics of the actual audio waveform itself as well.

Which Pipecat (open source voice AI orchestration platform) actually does as well seemingly with their smart-turn native turn detection model as well: https://github.com/pipecat-ai/smart-turn (minus the built-in transcription)

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#105

Post by medi8r » Tue, Mar 03, 2026, 6:39 AM UTC

Maybe we have a keyword to say we are done talking. Like "over to you". This may be better as it gives you thinking time.

Even a minute if you need it!

And you can get the agent to crunch when you are ready.

Imagine you speak. you need to look something up. find it. speak some more. then "over to you!"

The agent doesn't have to behave like a human and figure out when to butt in.

After all chat rooms and Slack also have realtime 2 way but we didn't worry about emulating that in agent chat. We can be convention breaking in agentic voice chat too.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#106

Post by mungoman2 » Tue, Mar 03, 2026, 6:44 AM UTC

jedberg wrote: post #36

Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…

I think you’re implying that it would be useful to have the LLM predict the end of the speaker’s speech, and continue with its reply based on that.

If, when the speaker actually stops speaking, there is a match vs predicted, the response can be played without any latency.

Seems like an awesome approach! One could imagine doing this prediction for the K most likely threads simultaneously, subject by computer power available, and prune/branch as some threads become inaccurate.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#107

Post by ilaksh » Tue, Mar 03, 2026, 7:06 AM UTC

alfalfasprout wrote: post #31

nicktikhonov wrote: post #19

Earlier quoted context omitted.

Very cool! starred and on my reading list. Would love to chat and share notes, if you'd like

Also consider using Cerebras' inference APIs. They released a voice demo a while back and the latency of their model inference is insane.

I tried to use Cerebras and it was unbeatable at first, but the client didn't want to pay $1300 a month and the $50/month or pay as you go was just not reliable. It would give service unavailable errors or falsely claim we were over our rate limit.

Also Groq is very fast, but the latency wasn't always consistent and I saw some very strange responses on a few calls that I had to attribute to quantization.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#108

Post by ilaksh » Tue, Mar 03, 2026, 7:11 AM UTC

nicktikhonov wrote: post #16

modeless wrote: post #14

IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy , but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.

If you're of that opinion, you'll enjoy the new stuff coming out from nvidia: https://research.nvidia.com/labs/adlr/personaplex/

It just about works for our current use case but can't comprehend the concept of an outgoing call. So I am trying to fine tune it. Tricky thing is personaplex forked some of the kyutai code and has not integrated the LoRA stuff they added. So we tried to do update personaplex with the fine tuning stuff. Going to find out tonight or tomorrow whether it's actually feasible when I finish debugg/testing.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#109

Post by ilaksh » Tue, Mar 03, 2026, 7:24 AM UTC

Just to mention, I have a similar solution on GitHub under my username runvnc, repo mindroot with plugins from repos mr_sip (should work with any SIP vendor although only tested on Telynx), mr_eleven_stream or mr_pocket-tts (which is free since it runs on CPU), and an LLM plugin like ah_openrouter, ah_anthropic or mr_gemini.

I also have a setting in mr_sip to use gpt-realtime via plugin ah_openai, which is very low latency speech-to-speech but quite expensive.

But my client saw the Sesame demo page, and so now I am trying to fine tune PersonaPlex.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#110

Post by russdill » Tue, Mar 03, 2026, 7:27 AM UTC

Barbing wrote: post #93

russdill wrote: post #86

Earlier quoted context omitted.

I've experimented with having different sized LLMs cooperating. The smaller LLM starts a response while the larger LLM is starting. It's fed the initial response so it can continue it. The idea of having an LLM follow and continuously predict the speaker. It would allow a response to be continually generated. If the prediction is correct, the response can be started with zero latency.

Google seems to be experimenting with this with their AI Mode. They used to be more likely to send 10 blue links in response to complex queries, but now they may instead start you off with slop. (Meanwhile at OpenAI: testing out the free ChatGPT, it feels like they prompted GPT 3.5 to write at length based on the last one or maybe two prompts)

This is more of a "Are all the windows closed upstairs?"

"The windows upstairs..."

"...are all closed except for the bedroom window"

The first portion of the response requires a couple of seconds to play but only a few tens of milliseconds to start streaming using a small model. Currently I just break the small model's response off at whatever point will produce about enough time to spin up the larger model.

But all responses spin up both models.