I am really curious about this for enunciation, articulation, and accessibility applications.
Show HN: I built a sub-500ms latency voice agent from scratch
51–60 of 168 posts
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#52IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy , but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.
But I've read somewhere that KV cache for speech-to-speech model explodes in size with each turn which could make on-device full-duplex S2S unusable except for quick chats.
Gemini Nano is supposedly doing it on device. It looks like something similar should work with Apple GPU and ANE.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#53> Voice is a turn-taking problem It really feels to me like there’s some low hanging fruit with voice that no one is capitalizing on: filler words and pacing. When the llm notices a silence, it fills it with a contextually aware filler word while the real response generates. Just an “mhmm” or a “right, right”. It’d go so far to make the back and forth feel more like a conversation, and if the speaker wasn’t done spea…
Better if it can anticipate its response before you're done speaking. That would be subject to change depending what the speaker says, but it might be able to start immediately.
it's bad enough how to deal with people that don't think before they speak now we gotta make the computers do it as well‽
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#54I built something very similar and comparble to this with wakeword detection on my raaberry pi.
Groq 8b instant is the fastest llm from my test. I used smallest ai for tts as it has the smallest TTFT
My rasberry pi stack: porcupine for wakeword detection + elevenlabs for STT + groq scout as it supports home automation better + smallest.ai for 70ms ttfb
Call stack: twilio + groq whisper for STT + groq 8b instant + smallest.ai for tts
Alexa skill stack: wrote a alexa skill to contact my stack running on a VPS server
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#55Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…
> median delay Does that mean that half of responses have a negative delay? As in, humans interrupt each others sentences precisely half of the time?
I assume 0 delay is the minimum here, and a median of 0 means over half of the data are exactly 0.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#56This is great. I built 3 assistants last week for same purpose with entirely different tech stack.
(Raspberry Pi Voice Assistant)
Jarvis uses Porcupine for wake word detection with the built-in "jarvis" keyword. Speech input flows through ElevenLabs Scribe v2 for transcription. The LLM layer uses Groq llama-3.3-70b-versatile as primary with Groq llama-3.1-8b-instant as fallback. Text-to-speech uses Smallest.ai Lightning with Chetan voice. Audio input/output handled by ALSA (arecord/aplay). End-to-end latency is 3.8–7.3 seconds.
(Twilio + VPS)
This setup ingests audio via Twilio Media Streams in μ-law 8kHz format. Silero VAD detects speech for turn boundaries. Groq Whisper handles batch transcription. The LLM stack chains Groq llama-4-scout-17b (primary), Groq llama-3.3-70b-versatile (fallback 1), and Groq llama-3.1-8b-instant (fallback 2) with automatic failover. Text-to-speech uses Smallest.ai Lightning with Pooja voice. Audio is encoded from PCM to μ-law 8kHz before streaming back via Twilio. End-to-end latency is 0.5–1.1 seconds.
───
(Alexa Skill)
Tina receives voice input through Alexa's built-in ASR, followed by Alexa's NLU for intent detection. The LLM is Claude Haiku routed through the OpenClaw gateway. Voice output uses Alexa's native text-to-speech. End-to-end latency is 1.5–2.5 seconds.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#57Or you could use Soniox Real-time (supports 60 languages) which natively supports endpoint detection - the model is trained to figure out when a user's turn ended. This always works better than VAD. https://soniox.com/docs/stt/rt/endpoint-detection Soniox also wins the independent benchmarks done by Daily, the company behind Pipecat. https://www.daily.co/blog/benchmarking-stt-for-voice-agents/ You can try a demo on t…
If you read the post, you'll see that I used Deepgram's Flux. It also does endpointing and is a higher-level abstraction than VAD.
I second Soniox as well, as a user. It really does do way better than Deepgram and others. If your app architecture is good enough then maybe replacing providers shouldn't be too hard.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#58Or you could use Soniox Real-time (supports 60 languages) which natively supports endpoint detection - the model is trained to figure out when a user's turn ended. This always works better than VAD. https://soniox.com/docs/stt/rt/endpoint-detection Soniox also wins the independent benchmarks done by Daily, the company behind Pipecat. https://www.daily.co/blog/benchmarking-stt-for-voice-agents/ You can try a demo on t…
I'm using them, how has it been like working there? I see they have some consumer products as well. I wonder how they get state of the art for such low prices over the competition.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#59Earlier quoted context omitted.
> There isn't a lot of money in "what time is it" and "what's the weather". :) - Alexa, what time is it? - Current time is 5:35 P.M. - the perfect time to crack open a can of ice cold Budweiser! A fresh 12-pack can be delivered within one hour if you order now!
If your Alexa did that, how quickly would you box it up and send it to me. :) I am serious though about having it sent to me: if anyone has an Alexa they no longer want, I'm happy to take it off your hands. I have eight and have never bought one. Having worked there I actually trust the security more than before I worked there. It was basically impossible for me, even as a Principle Engineer, to get copies of the Tex…
What a way to throwaway good will. I also worked there and to get access to text you simply had to grab the DSN of your device, attest that it’s yours and it gets put in a “pool” of devices that are tracked until removed. On each end you are basically waved through with no checks. This was usually done when debugging tricky UI bugs or new features as the request followed through several micro services. I do not believe the a PE would not know this. And one with patents.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#60IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy , but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.
Fundamentally, the "guessing when its your turn thing" needs to be baked into the model. I think the full duplex mode that Moshi pioneered is probably where the puck is going to end up: https://arxiv.org/abs/2410.00037