This was discussed 21 days ago: https://news.ycombinator.com/item?id=46946705
"extensively" = 2 comments?
You're right, fixed it. I discussed it extensively with a colleague and that got conflated.
It's a great article.
Show HN: I built a sub-500ms latency voice agent from scratch
41–50 of 168 posts
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#42Earlier quoted context omitted.
This is fascinating, thanks for sharing! I wonder why amazon/google/apple didn't hop on the voice assistant/agent train in the last few years. All 3 have existing products with existing users and can pretty much define and capture the category with a single over-the-air update.
Two main reasons: 1. Compute. It's easy to make a voice assistant for a few people. But it takes a hell of a lot of GPU to serve millions. 2. Guard Rails. All of those assistants have the ability to affect the real world. With Alexa you can close a garage or turn on the stove. It would be real bad if you told it to close the garage as you went to bed for the night and instead it turned on the stove and burned down th…
> There isn't a lot of money in "what time is it" and "what's the weather". :)
- Alexa, what time is it?
- Current time is 5:35 P.M. - the perfect time to crack open a can of ice cold Budweiser! A fresh 12-pack can be delivered within one hour if you order now!
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#43Earlier quoted context omitted.
Two main reasons: 1. Compute. It's easy to make a voice assistant for a few people. But it takes a hell of a lot of GPU to serve millions. 2. Guard Rails. All of those assistants have the ability to affect the real world. With Alexa you can close a garage or turn on the stove. It would be real bad if you told it to close the garage as you went to bed for the night and instead it turned on the stove and burned down th…
> There isn't a lot of money in "what time is it" and "what's the weather". :) - Alexa, what time is it? - Current time is 5:35 P.M. - the perfect time to crack open a can of ice cold Budweiser! A fresh 12-pack can be delivered within one hour if you order now!
If your Alexa did that, how quickly would you box it up and send it to me. :)
I am serious though about having it sent to me: if anyone has an Alexa they no longer want, I'm happy to take it off your hands. I have eight and have never bought one. Having worked there I actually trust the security more than before I worked there. It was basically impossible for me, even as a Principle Engineer, to get copies of the Text to Speech of a customer and I literally never heard a customer voice recording.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#44IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy , but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.
But I've read somewhere that KV cache for speech-to-speech model explodes in size with each turn which could make on-device full-duplex S2S unusable except for quick chats.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#45[dead]
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#46> Voice is a turn-taking problem It really feels to me like there’s some low hanging fruit with voice that no one is capitalizing on: filler words and pacing. When the llm notices a silence, it fills it with a contextually aware filler word while the real response generates. Just an “mhmm” or a “right, right”. It’d go so far to make the back and forth feel more like a conversation, and if the speaker wasn’t done spea…
100% - I thought about that shortly after writing this up. One way to make this work is to have a tiny, lower latency model generate that first reply out of a set of options, then aggressively cache TTS responses to get the latency super low. Responses like "Hmm, let me think about that..." would be served within milliseconds.
Years ago I wrote a system that would generate Lucene queries on the fly and return results. The ~250 ms response time was deemed too long, so I added some information about where the response data originated, and started returning "According to..." within 50 ms of the end of user input. So the actual information got to the user after a longer delay, but it felt almost as fast as conversion.
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#47[flagged]
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#48Re: Show HN: I built a sub-500ms latency voice agent from scratch
#49Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…
> median delay
Does that mean that half of responses have a negative delay? As in, humans interrupt each others sentences precisely half of the time?
Re: Show HN: I built a sub-500ms latency voice agent from scratch
#50When someone is able to put something like this together on their own it leaves me feeling infuriated that we can’t have nice things on consumer hardware.
At a minimum Siri, Alexa, and Google Home should at least have a path to plugin a tool like this. Instead I’m hacking together conversation loops in iOS Shortcuts to make something like this style of interaction with significantly worse UX.