Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

151–160 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#151

Post by Dr4kn »

TeMPOraL wrote: post #147
riquito wrote: post #111

Earlier quoted context omitted.

Very cool, thanks for sharing. A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?

FWIW, wake words are a stopgap; if we want to have a Star Trek level voice interfaces, where the computer responds only when you actually meant to call it, as opposed to using the wake word as a normal word in the conversation, the computer needs to be constantly listening. A good analogy here is to think of the computer (assistant) as another person in the room, busy with their own stuff but paying attention to the…

Home Assistant is much nearer to this than other solutions.

You have a wake word, but it can also speak to you based on automations. You come home and it could tell you that the milk is empty, but with a holiday coming up you probably should go shopping.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#152

Post by smusamashah »

randomcatuser wrote: post #116
smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

yeah the demo I saw was: https://x.com/livekit/status/1870194686532694417 But searching for "voice detection with pauses", it seems there's a lot of new contenders! https://x.com/kwindla/status/1897711929617154148 this one is a fun approach too https://x.com/zan2434/status/1753660774541849020

This is the one I saw https://x.com/kwindla/status/1870974144831275410

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#153

Post by supermatt »

Dr4kn wrote: post #150
justlikereddit wrote: post #144

Earlier quoted context omitted.

Modify it with an ultra light LLM agent that always listens that uses a wake word to agentically call the paid API?

You could use open wake word. Which Home Assistant developed for its own Voice Assistant

It was developed by David Scripka: https://github.com/dscripka/openWakeWord

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#154

Post by VagabundoP »

Quizzical4230 wrote: post #134
joshstrange wrote: post #27

Earlier quoted context omitted.

This 100%, yes! I've found myself putting in filler words or holding a noise "Uhhhhhhhhh" while I'm trying to form a thought but I don't want the LLM to start replying. It's a really hard problem for sure. Similar to the problem of allowing for interruptions but not stopping if the user just says "Right!", "Yes", aka active listening. One thing I love about MacWhisper (not special to just this STT tool) is it's hold…

I recently got to know about this[^1] paper that differentiates between 'uh' and 'um' . > The proposal examined here is that speakers use uh and um to announce that they are initiating what they expect to be a minor (uh), or major (um), delay in speaking. Speakers can use these announcements in turn to implicate, for example, that they are searching for a word, are deciding what to say next, want to keep the floor, o…

I hate when you get "out of sync" with someone for a whole conversation. I imagine sine ways on an occilloscope and there they just slightly out of phase.

You nearly have to do a hard reset to get things comforatble - walk out of the room, ring the back.

But some people are just out of sync with the world.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#155

Post by regularfry »

nitrogen99 wrote: post #92

Will this work on a Raspberry Pi?

Not reliably. It can only drive Whisper quickly enough to appear real-time because of the GPU, and without that you're limited to the tiny/small/base models to get latency into single-digit seconds.

Edit to add: this might not be true since whisper-large-v3-turbo got released. I've not tried that on a pi 5 yet.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#156

Post by regularfry »

We really, really need something to take Whisper's crown for streaming. Faster-whisper is great, but Whisper itself was never built for real-time use.

For this demo to be real-time, it relies on having a beefy enough GPU that it can push 30 seconds of audio through one of the more capable (therefore bigger) models in a couple of hundred milliseconds. It's basically throwing hardware at the problem to paper over the fact that Whisper is just the wrong architecture.

Don't get me wrong, it's great where it's great, but that's just not streaming.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#159

Post by Dlemo »

TeMPOraL wrote: post #147
riquito wrote: post #111

Earlier quoted context omitted.

Very cool, thanks for sharing. A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?

FWIW, wake words are a stopgap; if we want to have a Star Trek level voice interfaces, where the computer responds only when you actually meant to call it, as opposed to using the wake word as a normal word in the conversation, the computer needs to be constantly listening. A good analogy here is to think of the computer (assistant) as another person in the room, busy with their own stuff but paying attention to the…

I want that for privacy reasons and for resource reasons.

And having this as a small hardware device should not add relevant latency to it.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#160

Post by koljab »

riquito wrote: post #111
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Very cool, thanks for sharing. A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?

That would be quite easy to integrate. RealtimeSTT already has wakeword support for both pvporcupine and openwakewords.
Post reply on HN