Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built a sub-500ms latency voice agent from scratch

ntik.me

Post reply on HN
Print view View original

121–130 of 168 posts

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#121

Post by nicktikhonov »

bronco21016 wrote: post #50

When someone is able to put something like this together on their own it leaves me feeling infuriated that we can’t have nice things on consumer hardware. At a minimum Siri, Alexa, and Google Home should at least have a path to plugin a tool like this. Instead I’m hacking together conversation loops in iOS Shortcuts to make something like this style of interaction with significantly worse UX.

I feel like you could get pretty far with a raspberry pi and microphone/speaker. I think the hard part is running a model that can detect a "Hey agent" on-device, so that it can run 24/7 and hand off to the orchestrator when it catches a real question/query.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#122

Post by nicktikhonov »

medi8r wrote: post #105

Maybe we have a keyword to say we are done talking. Like "over to you". This may be better as it gives you thinking time. Even a minute if you need it! And you can get the agent to crunch when you are ready. Imagine you speak. you need to look something up. find it. speak some more. then "over to you!" The agent doesn't have to behave like a human and figure out when to butt in. After all chat rooms and Slack also ha…

One thing you can get the LLM to do is to call a "skip turn" tool, which will basically trigger the system to wait without saying anything. Then all it will take is clever prompting to get the desired result.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#125

Post by nicktikhonov »

kelvinjps10 wrote: post #61

The quality of the post was amazing, I'm not that interested into voice agents yet but that I was engaged in the whole post. And the little animation made it easier to understand the loop.

Glad to hear! I built my blog on top of NextJS - it basically just renders .mdx files with contentlayer. One of the things I discovered is that you can easily vibe-code these explainer widgets. Seems like a perfect use case for vibe coding - each is a simple react component and I can keep iterating until I get it working just the way I like. And super easy to interleave with content. Seems like this could be an obvious feature addition to all the blogging platforms.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#127

Post by nicktikhonov »

hosaka wrote: post #71

Depending on the TTS model being used latency can be reduced further yet with an LRU cache, fetching common phrases from cache instead of generating fresh with TTS. However the naturalness of how it sounds will depend on how the TTS model works and whether two identical chunks of text will sound alike every generation.

Yep. Seems like caching more broadly is something worth exploring next if I were to do a pt2.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#128

Post by layer8 »

jedberg wrote: post #36

Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…

I disagree with fact 2, voice assistant latency is annoyingly slow. It often causes a conscious wait like “did it work or did it not?”. Cell phone delay is bad as well, it’s certainly not an expectation that carries over to other devices for me.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#129

Post by layer8 »

iso1631 wrote: post #113
miki_oomiri wrote: post #111

Earlier quoted context omitted.

When I speak to an agent, siri, or whatnot, I am always worried that they will assume I'm done talking when I'm thinking. Sometimes I need a many-seconds pause. Even maybe a minute… For Sire and such, I want to ask something simple "Hey Siri, remind me to call dad tomorrow". Easy. But for Claude and such, I want to go on a long monolog (20s, a minute, multi-minutes). To me, be the best solution would be semantic + ke…

And suddenly your address book has changed the name from "Dad" to "Tomorow"

Never skip an opportunity for a dad joke.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#130

Post by evara-ai »

Great writeup. I've been building production voice agents and automation systems with the same stack (Twilio + Deepgram + ElevenLabs + LLM APIs) for client-facing use cases — appointment booking, lead qualification, guest concierge, and workflow orchestration.

The "turn-taking problem, not transcription problem" framing is exactly right. We burned weeks early on optimizing STT accuracy when the actual UX killer was the agent jumping in mid-sentence or waiting too long. Switching from fixed silence thresholds to semantic end-of-turn detection was night and day.

One dimension I'd add: geography matters even more when your callers are in a different region than your infrastructure. We serve callers in India connecting to US-East, and the Twilio edge hop alone adds 150-250ms depending on the carrier. Region-specific deployments with caller-based routing helped a lot.

The barge-in teardown is the part most people underestimate. It's not just canceling LLM + TTS — if you have downstream automation (updating booking state, triggering webhook workflows, writing to DB), you need to handle the race condition where the system already committed to a response path that's now invalid. We had a bug where a barged-in appointment confirmation was still triggering the downstream booking pipeline.

Post reply on HN