I had been working on something like it when I came across this. Excellent work. Love the demo.
Show HN: Real-time AI Voice Chat at ~500ms Latency
81–90 of 238 posts
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#82This kind of thing immediately made me think about the 512gb mac studio. If this works as good on that hardware as it does on the recommended nvidia cards, then the $15k is not much the price of the hardware but rather the price of having a full conversational at home, private.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#83Earlier quoted context omitted.
> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning. I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multi…
You're right, this is not solvable with regular LLMs. It's not possible to mimic natural conversational rhythm with a separate LLM generating text, a separate text-to-speech generating audio, and a separate VAD determining when to respond and when to interrupt. I strongly believe you have to do everything in one model to solve this issue, to let the model decide when to speak, when to interrupt the user even. The onl…
Whoah, how odd. It asked me what I was doing, I said I just ate a burger. It then got really upset about how hungry it is but is unable to eat and was unable to focus on other tasks because it was “too hungry”. Wtf weirdest LLM interaction I’ve had.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#84I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…
A lot better techniques exist now days than pure silence detection - 1. A special model that predicts when a conversation turn is coming up (e.g. when someone is going to stop speaking). Speech has a rhythm to it and pauses / ends of speech are actually predictable. 2. Generate a model response for every subsequent word that comes in (and throw away the previously generated response), so basically your time to speak…
I think 2 & 3 should be combined. The AI should just finish the current sentence (internally) before it's being spoken, and once it reaches a high enough confidence, stick with the response. That's what humans do, too. We gather context and are able to think of a response while the other person is still talking.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#85This is great. Poking into the source, I find it interesting that the author implemented a custom turn detection strategy, instead of using Silero VAD (which is standard in the voice agents space). I’m very curious why they did it this way and what benefits they observed. For folks that are curious about the state of the voice agents space, Daily (the WebRTC company) has a great guide [1], as well as an open-source f…
It's in fact using Silero via RealtimeSTT. RealtimeSTT tells when silence starts. Then a binary sentence classification model is used on the realtime transcription text which infers blazingly fast (10ms) and returns a probability between 0 and 1 indicating if the current spoken sentence is considered "complete". The turn detection component takes this information to calculate the silence waiting time until "turn is over".
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#86I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…
> The median delay between speakers in a human to human conversation is zero milliseconds
What about on phone calls? When I'm on a call with customer support they definitely wait for it to be clear that I'm done talking before responding, just like AI does.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#87I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…
This feels intuitively correct to me, although I am more informed as an audio engineer than a software/LLM one. That said, is ~500ms considered “real-time” in this context? I’ve worked on recording workflows, and it’s basically geologic time in that context.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#88After interrupt, unspoken words from LLM are still in the chat window. Is LLM even aware that it was interrupted and where exactly?
It's not aware. The information that it had been interrupted would be something we can easily add to the next user chat request. Where exactly is harder, because at least for Coqui XTTSv2 we don't have TTS wordstamps (we do have them for Kokoro though).
So adding the information where it had been interrupted would be easily possible when using Kokoro as TTS system. With Coqui we'd need to add another transcription on the tts output including word timestamps. That would cost more compute than a normal transcription and word timestamps aren't perfectly accurate. Yet directly after an interruption there's not that much concurrent need for compute (like in the end of turn detection phase where a lot of stuff is happening). So I guess with a bit of programming work this could be integrated.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#89Call me when the AI can interrupt YOU :)
In Soviet Russia...
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#90Kind of surprised nobody has brought up https://www.sesame.com/research/crossing_the_uncanny_valley_... It interacts nearly like a human, can and does interrupt me once it has enough context in many situations, and has exceedingly low levels of latency, using for the first time was a fairly shocking experience for me.
Didn't expect it to be that good! Nice.
Yeah, thats one of the best ones I have seen, and it popped up a while ago.