Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

111–120 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#111

Post by riquito »

koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Very cool, thanks for sharing.

A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#112

Post by wkat4242 »

kadushka wrote: post #79
varispeed wrote: post #52

Earlier quoted context omitted.

> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

Read this if you haven’t already: https://marshallbrain.com/manna1 That’s a much more serious anxiety trigger for me.

Me too. Thanks for that, didn't know about it.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#113

Post by jyoung8607 »

mvdtnz wrote: post #31
WhitneyLand wrote: post #16

Earlier quoted context omitted.

>>they don't allow talking naturally Neither do phone calls. Round trip latency can easily be 300ms, which we’ve all learned to adapt our speech to. If you want to feel true luxury find an old analog PTSN line. No compression artifacts or delays. Beautiful and seamless 50ms latency. Digital was a terrible event for call quality.

I don't know how your post is relevant to the discussion of AI models interrupting if I pause for half a second?

It's genuinely a very similar problem. The max round trip latency before polite humans start having trouble talking over each other has been well studied since the origins of the Bell Telephone system. IIRC we really like it to be under about 300ms.

AI has processing delay even if run locally. In telephony the delays are more speed-of-light dictated. But the impact on human interactive conversation is the same.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#114

Post by echelon »

koljab wrote: post #41
echelon wrote: post #30

Earlier quoted context omitted.

Are you using all local models, or does it also use cloud inference? Proprietary models? Which models are running in which places? Cool utility!

All local models: - VAD: Webrtcvad (first fast check) followed by SileroVAD (high compute verification) - Transcription: base.en whisper (CTranslate2) - Turn Detection: KoljaB/SentenceFinishedClassification (selftrained BERT-model) - LLM: hf.co/bartowski/huihui-ai_Mistral-Small-24B-Instruct-2501-abliterated-GGUF:Q4_K_M (easily switchable) - TTS: Coqui XTTSv2, switchable to Kokoro or Orpheus (this one is slower)

That's excellent. Really amazing bringing all of these together like this.

Hopefully we get an open weights version of Sesame [1] soon. Keep watching for it, because that'd make a killer addition to your app.

[1] https://www.sesame.com/

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#115

Post by [deleted] »

clbrmbr wrote: post #83
peturdarri wrote: post #74

Earlier quoted context omitted.

You're right, this is not solvable with regular LLMs. It's not possible to mimic natural conversational rhythm with a separate LLM generating text, a separate text-to-speech generating audio, and a separate VAD determining when to respond and when to interrupt. I strongly believe you have to do everything in one model to solve this issue, to let the model decide when to speak, when to interrupt the user even. The onl…

Whoah, how odd. It asked me what I was doing, I said I just ate a burger. It then got really upset about how hungry it is but is unable to eat and was unable to focus on other tasks because it was “too hungry”. Wtf weirdest LLM interaction I’ve had.

[deleted]

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#116

Post by randomcatuser »

smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

yeah the demo I saw was: https://x.com/livekit/status/1870194686532694417

But searching for "voice detection with pauses", it seems there's a lot of new contenders!

https://x.com/kwindla/status/1897711929617154148

this one is a fun approach too https://x.com/zan2434/status/1753660774541849020

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#119

Post by hegemon8 »

This is an impressive project—great work! I’m curious anyone has came across similar work, but for multi-lingual voice agents, especially those that handle non-English languages and English + X well.

Does a Translation step right after the ASR step make sense at all?

Any pointers—papers, repos —would be appreciated!

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#120

Post by flippy_flops »

qwertox wrote: post #5
smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

Maybe we should settle on some special sound or word which officially signals that we're making a pause for whatever reason, but that we intend to continue with dictating in a couple of seconds. Like "Hmm, wait".

Need some vocal version of “heredoc”
Post reply on HN