Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

51–60 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#51

Post by tomp »

joshstrange wrote: post #42
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning. I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multi…

If your model is fast enough, you can definitely do it. That's literally how "streaming Whisper" works, just rerun the model on the accumulated audio every x00ms. LLMs could definitely work the same way, technically they're less complex than Whisper (which is an encoder/decoder architecture, LLMs are decoder-only) but of course much larger (hence slower), so ... maybe rerun just a part of it? etc.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#52

Post by varispeed »

robbomacrae wrote: post #45
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

Spot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.

> AI in KFC drive thrus

That right here is an anxiety trigger and would make me skip the place.

There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#53

Post by modeless »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

My take on this is that voice AI has not truly arrived until it has mastered the "Interrupting Cow" benchmark.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#54

Post by briga »

I'm starting to feel like LLMs need to be tuned for shorter responses. For every short sentence you give them they outputs paragraphs of text. Sometimes it's even good text, but not every input sentence needs a mini-essay in response.

Very cool project though. Maybe you can fine tune the prompt to change how chatty your AI is.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#56

Post by smeej »

krainboltgreene wrote: post #44
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

I would also suspect that a human has much less patience for a robot interrupting them than a human.

I'm certainly in that category. At least with a human, I can excuse it by imagining the person grew up with half a dozen siblings and always had to fight to get a word in edgewise. With a robot, it's interrupting on purpose.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#57

Post by koljab »

kevinsync wrote: post #38
PhunkyPhil wrote: post #24

Earlier quoted context omitted.

This is the system prompt https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/s... My favorite line: "You ARE this charming, witty, wise girlfriend. Don't explain how you're talking or thinking; just be that person."

I still crack up at the idea of 'personality prompting', mostly because the most engaging and delightful IRL persons who knock us off our guard in a non-threatening way are super natural and possess that "It Factor" that's impossible to articulate lol -- probably because it's multimodal with humans and voice/cadence/vocab/timing/delivery isn't 100% of the attraction. That said, it's not like we have any better altern…

This character prompt has undergone so many iterations with LLMs it's not funny anymore. "Make her act more bold." - "She again talked about her character description, prevent that!"

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#58

Post by coolspot »

varispeed wrote: post #52
robbomacrae wrote: post #45

Earlier quoted context omitted.

Spot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.

> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

They have a fallback to a human operator when stopwords and/or stop conditions are detected.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#59

Post by varispeed »

cannonpr wrote: post #49

Kind of surprised nobody has brought up https://www.sesame.com/research/crossing_the_uncanny_valley_... It interacts nearly like a human, can and does interrupt me once it has enough context in many situations, and has exceedingly low levels of latency, using for the first time was a fairly shocking experience for me.

Didn't expect it to be that good! Nice.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#60

Post by varispeed »

kevinsync wrote: post #38
PhunkyPhil wrote: post #24

Earlier quoted context omitted.

This is the system prompt https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/s... My favorite line: "You ARE this charming, witty, wise girlfriend. Don't explain how you're talking or thinking; just be that person."

I still crack up at the idea of 'personality prompting', mostly because the most engaging and delightful IRL persons who knock us off our guard in a non-threatening way are super natural and possess that "It Factor" that's impossible to articulate lol -- probably because it's multimodal with humans and voice/cadence/vocab/timing/delivery isn't 100% of the attraction. That said, it's not like we have any better altern…

Aren't humans doing it as well? It's called affirmations. Many people do this as their morning "boot" time.
Post reply on HN