Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

91–100 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#91

Post by lotyrin »

varispeed wrote: post #52
robbomacrae wrote: post #45

Earlier quoted context omitted.

Spot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.

> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

My AI drive thru experiences have been vastly superior to my human ones. I know it's powered by LLM and some kind of ability to parse my whole sentence (paying attention the whole time) and then it can key in whatever I said all at once.

With a human, I have to anticipate what order their POS system allows them to key things in, how many things I can buffer up with them in advance before they overflow and say "sorry, what size of coke was that, again", whether they prefer me to use the name of the item or the number of the item (based on what's easier to scan on the POS system). Because they're fatigued and have very little interest or attention to provide, having done this repetitive task far too many times, and too many times in a row.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#93

Post by jimbokun »

Reason077 wrote: post #64
jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

The best, most human-like AI voice chat I've seen yet is Sesame (www.sesame.com). It has delays, but fills them very naturally with normal human speech nuances like "hmmm", "uhhh", "hold on while I look that up" etc. If there's a longer delay it'll even try to make a bit of small talk, just like a human conversation partner might.

When learning Japanese in Japan, I figured out one way to sound more native was to just add interjections like “Eeee?” (really?) and “Sou desu ka?” (is that so?) while the other person was talking. Makes it sound like you are paying attention and following what they are saying.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#94

Post by jimbokun »

clbrmbr wrote: post #83
peturdarri wrote: post #74

Earlier quoted context omitted.

You're right, this is not solvable with regular LLMs. It's not possible to mimic natural conversational rhythm with a separate LLM generating text, a separate text-to-speech generating audio, and a separate VAD determining when to respond and when to interrupt. I strongly believe you have to do everything in one model to solve this issue, to let the model decide when to speak, when to interrupt the user even. The onl…

Whoah, how odd. It asked me what I was doing, I said I just ate a burger. It then got really upset about how hungry it is but is unable to eat and was unable to focus on other tasks because it was “too hungry”. Wtf weirdest LLM interaction I’ve had.

Damn they trained a model that so deeply embeds human experience it actually feels hunger, yet self aware enough it knows it’s not capable of actually eating!

That’s like a Black Mirror episode come to life.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#95

Post by [deleted] »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

[deleted]

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#96

Post by peterldowns »

koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models

In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#97

Post by dankwizard »

zaggynl wrote: post #19
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Neat! I'm already using openwebui/ollama with a 7900 xtx but the STT and TTS parts don't seem to work with it yet: 2025-05-05 20:53:15,808] [WARNING] [real_accelerator.py:194:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it. Error loading model for checkpoint ./models/Lasinya: This op had not been implemented on CPU backend.

I've given up trying to locally use LLMs on AMD

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#98

Post by kibibu »

varispeed wrote: post #52
robbomacrae wrote: post #45

Earlier quoted context omitted.

Spot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.

> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

Are we entering a new era of KFC drive-through jailbreaks?

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#99

Post by mmoustafa »

smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

Huge problem space. Usually referred to as “turn taking”

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#100

Post by jiehong »

joshstrange wrote: post #27
smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

This 100%, yes! I've found myself putting in filler words or holding a noise "Uhhhhhhhhh" while I'm trying to form a thought but I don't want the LLM to start replying. It's a really hard problem for sure. Similar to the problem of allowing for interruptions but not stopping if the user just says "Right!", "Yes", aka active listening. One thing I love about MacWhisper (not special to just this STT tool) is it's hold…

So they basically train us to worsen our speech to avoid being interrupted.

I remember my literature teacher telling us in class how we should avoid those filler words, and instead allow for some simple silences while thinking.

Although, to be fair, there are quite a few people in real life using long filler words to avoid anyone interfering them, and it’s obnoxious.

Post reply on HN