Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

41–50 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#41

Post by koljab »

echelon wrote: post #30
koljab wrote: post #13

Earlier quoted context omitted.

I only tested it on my 4090 so far

Are you using all local models, or does it also use cloud inference? Proprietary models? Which models are running in which places? Cool utility!

All local models: - VAD: Webrtcvad (first fast check) followed by SileroVAD (high compute verification) - Transcription: base.en whisper (CTranslate2) - Turn Detection: KoljaB/SentenceFinishedClassification (selftrained BERT-model) - LLM: hf.co/bartowski/huihui-ai_Mistral-Small-24B-Instruct-2501-abliterated-GGUF:Q4_K_M (easily switchable) - TTS: Coqui XTTSv2, switchable to Kokoro or Orpheus (this one is slower)

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#42

Post by joshstrange »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning.

I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multiple generations internally and starts generating after every word, or at least starts asking sub-LLM models "Do I have enough to reply?" and once it does it generates a reply and interrupts.

I'm not sure how most apps handle the user interrupting, in regards to the conversation context. Do they stop generation but use what they have generated already in the context? Do they cut off where the LLM got interrupted? Something like "LLM: ..and then the horse walked... -USER INTERRUPTED-. User: ....". It's not a purely-voice-LLM issue but it comes up way more for that since rarely are you stopping generation (in the demo, that's been done for a while when he interrupts), just the TTS.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#43

Post by jedberg »

koljab wrote: post #39
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

Thanks a lot, great insights. Exactly the kind of feedback that I need to improve things further.

Love what you're doing, glad I could help!

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#44

Post by krainboltgreene »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

I would also suspect that a human has much less patience for a robot interrupting them than a human.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#45

Post by robbomacrae »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

Spot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#48

Post by r0fl »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

Great insights. When I have a conversation with another person sometimes they cut me off when they are trying to make a point. I have talked to ChatGPT and grok at length (hours of brain storming, learning things, etc) and AI has never interrupted aggressively to try to make a point stick better

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#49

Post by cannonpr »

Kind of surprised nobody has brought up https://www.sesame.com/research/crossing_the_uncanny_valley_...

It interacts nearly like a human, can and does interrupt me once it has enough context in many situations, and has exceedingly low levels of latency, using for the first time was a fairly shocking experience for me.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#50

Post by varispeed »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

This silence detection is what makes me unable to chat with AI. It is not natural and creates pressure.

True AI chat should know when to talk based on conversation and not things like silence.

Voice to text is stripping conversation from a lot of context as well.

Post reply on HN