Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

71–80 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#71

Post by wyager »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

> The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative.

Fascinating. I wonder if this is some optimal information-theoretic equilibrium. If there's too much average delay, it means you're not preloading the most relevant compressed context. If there's too little average delay, it means you're wasting words.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#73

Post by dummydummy1234 »

koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Have you looked at pipecat, seems to be similar trying to do standardized backend/webrtc turn detection pipelines.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#74

Post by peturdarri »

joshstrange wrote: post #42
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning. I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multi…

You're right, this is not solvable with regular LLMs. It's not possible to mimic natural conversational rhythm with a separate LLM generating text, a separate text-to-speech generating audio, and a separate VAD determining when to respond and when to interrupt. I strongly believe you have to do everything in one model to solve this issue, to let the model decide when to speak, when to interrupt the user even.

The only model that has attempted this (as far as I know) is Moshi from Kyutai. It solves it by having a fully-duplex architecture. The model is processing the audio from the user while generating output audio. Both can be active at the same time, talking over each other, like real conversations. It's still in research phase and the model isn't very smart yet, both in what it says and when it decides to speak. It just needs more data and more training.

https://moshi.chat/

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#75

Post by snet0 »

koljab wrote: post #29
joshstrange wrote: post #25

This is very, very cool! The interrupting was a "wow" moment for me (I know it's not "new new" but to see it so well done in open source was awesome). Question about the Interrupt feature, how does it handle "Mmk", "Yes", "Of course", " cough ", etc? Aside from the sycophancy from OpenAI's voice chat (no, not every question I ask is a "great question!") I dislike that a noise sometimes stops the AI from responding an…

That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming realtime transcription as a trigger. That adds a bit of latency but that gets compensated by way better accuracy. Edit: just realized the irony but it's really a good question lol

> That's a great question!

Never forget what AI stole from us. This used to be a compliment, a genuine appreciation of a good question well-asked. Now it's tainted with the slimy, servile, sycophantic stink of AI chat models.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#76

Post by lelandfe »

Reason077 wrote: post #64
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

The best, most human-like AI voice chat I've seen yet is Sesame (www.sesame.com). It has delays, but fills them very naturally with normal human speech nuances like "hmmm", "uhhh", "hold on while I look that up" etc. If there's a longer delay it'll even try to make a bit of small talk, just like a human conversation partner might.

So-called backchanneling https://wikipedia.org/wiki/Backchannel_(linguistics)

> The person doing the speaking is thought to be communicating through the "front channel" while the person doing the listening is thought to be communicating through the "backchannel”

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#77

Post by com2kid »

joshstrange wrote: post #42
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning. I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multi…

Been there, implemented it, it works well enough.

Better solutions are possible but even tiny models are capable of being given a partial sentence and replying with a probability that the user is done talking.

The linked repo does this, it should work fine.

More advanced solutions are possible (you can train a model that does purely speech -> turn detection probability w/o an intermediate text step), but what the repo does will work well enough for many scenarios.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#78

Post by bigmadshoe »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

"The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative."

Is that really a productive way to frame it? I would imagine there is some delay between one party hearing the part of the sentence that triggers the interruption, and them actually interrupting the other party. Shouldn't we quantify this?

I totally agree that the fact the AI doesn't interrupt you is what makes it seem non-human. Really, the models should have an extra head that predicts the probability of an interruption, and make one if it seems necessary.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#79

Post by kadushka »

varispeed wrote: post #52
robbomacrae wrote: post #45

Earlier quoted context omitted.

Spot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.

> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

Read this if you haven’t already: https://marshallbrain.com/manna1

That’s a much more serious anxiety trigger for me.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#80

Post by elmomle »

bigmadshoe wrote: post #78
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

"The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative." Is that really a productive way to frame it? I would imagine there is some delay between one party hearing the part of the sentence that triggers the interruption, and them actually interrupting the other party. Shouldn't we quantif…

"Necessary" is an interesting framing. Here are a few others:

- Expeditious - Constructive - Insightful -

Post reply on HN