Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built a sub-500ms latency voice agent from scratch

ntik.me

Post reply on HN
Print view View original

141–150 of 168 posts

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#141

Post by GrayShade »

mywacaday wrote: post #136
jedberg wrote: post #36

Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…

No. 2 is interesting, our national lottery in Ireland has an app that you can scan the barcode on your ticket to check if you have won or not, at some stage they updated the app and the scan picks up the barcode even before you center it on the screen and tells you if you have lost/won instantly, I though it was my IT background that made me uncomfortable with it happening so fast, wonder what other examples like thi…

The Signal device linking feature is just as fast. It's partly a trick -- it will look for QR codes even outside the central area, so under good conditions it can get a read before you even get a rough orientation.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#142

Post by bachittle »

I'm running a local voice agent on a Mac Mini M4. Qwen ASR for STT and Qwen TTS on Apple Silicon via MLX, Claude for the LLM. No API costs besides the Claude subscription but the interesting part is the LLM is agentic because it's using Claude Code. It reads and writes files, spawns background agents, controls devices, all through voice.

The insights about VAD and streaming pipelines in this thread are exactly what I'm looking at for v2. Moving to a WebSocket streaming pipeline with proper voice activity detection would close the latency gap significantly, even with local models.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#144

Post by vidarh »

jedberg wrote: post #43
mcbits wrote: post #42

Earlier quoted context omitted.

> There isn't a lot of money in "what time is it" and "what's the weather". :) - Alexa, what time is it? - Current time is 5:35 P.M. - the perfect time to crack open a can of ice cold Budweiser! A fresh 12-pack can be delivered within one hour if you order now!

If your Alexa did that, how quickly would you box it up and send it to me. :) I am serious though about having it sent to me: if anyone has an Alexa they no longer want, I'm happy to take it off your hands. I have eight and have never bought one. Having worked there I actually trust the security more than before I worked there. It was basically impossible for me, even as a Principle Engineer, to get copies of the Tex…

I already swear at mine when it tries to suggest setting up a routine for me or otherwise fail to just immediately shut up after answering my query.

Still not boxing them up. Though I now have a Pi with a HomeAssistant setup I'm trialling, so maybe that'll change.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#145

Post by red2awn »

What's the SOTA open source or weight available turn taking model these days? I tried pipecat/smart-turn-v3 and the results are not good. It only works well when you say a short sentence in a clear voice. Anything else will cause it to wait indefinitely. Closed source API models are obviously a lot better but adds network latency and the cost adds up.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#146

Post by jermaustin1 »

sinuhe69 wrote: post #94
emmelaich wrote: post #78

Earlier quoted context omitted.

Regarding 2, I believe that talking on mobile phones drives older people crazy. They remember talking on normal land lines when there was almost no latency at all. The thing is -- they don't know why they don't like it.

Yeah, I remember the time when we had to use satellites to connect. The long delay was really annoying and so unusual that most people without "training" could not even use the phone for conversation and just wasted the dollars.

A former boss of mine took off to Everest for a month leaving me (a 22 year old, at the time) in charge of the office. I was out to dinner with my now wife when I got a call from a very long phone number I didn't recognize, so I ignored it. I then got another one right after, and picked it up. It was my boss, he needed me to log into his personal email to grab a phone number for the medical insurance he purchased for the trip, because he had been vomiting for days due to altitude sickness, and needed a medical evacuation.

That was the most stressfully hard to use phone call I've ever had. The delay was nearly 10 seconds, and eventually I just said I was only going to speak yes or no, if he needed a longer answer he needed to shut up. And that worked. We no longer talked over eachother.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#150

Post by totetsu »

cootsnuck wrote: post #104
totetsu wrote: post #99

Earlier quoted context omitted.

What is the difference between Flux’s end-of-turn detection and Openai's Automatic turn detection Semantic mode?

In OpenAI's own words about semantic_vad: > Chunks the audio when the model believes based on the words said by the user that they have completed their utterance. Source: https://developers.openai.com/api/docs/guides/realtime-vad OpenAI's Semantic mode is looking at the semantic meaning of the transcribed text to make an educated guess about where the user's end of utterance is. According to Deepgram, Flux's end-of-t…

Thanks. Then maybe it’s similar to Moshi https://github.com/kyutai-labs/moshi?tab=readme-ov-file
Post reply on HN