Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

141–150 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#141

Post by optimog »

peterldowns wrote: post #96
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

Seems like they are out of business. Their homepage mentions "Coqui is schutting down"* That is probably the reason you can't find that much.

*https://coqui.ai/

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#144

Post by justlikereddit »

riquito wrote: post #111
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Very cool, thanks for sharing. A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?

Modify it with an ultra light LLM agent that always listens that uses a wake word to agentically call the paid API?

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#145

Post by Tokumei-no-hito »

kadushka wrote: post #79
varispeed wrote: post #52

Earlier quoted context omitted.

> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

Read this if you haven’t already: https://marshallbrain.com/manna1 That’s a much more serious anxiety trigger for me.

wow that was incredible. thank you for sharing it. why does it cause you anxiety?

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#146

Post by pzo »

koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

This looks great will definitely have a look. I'm just wondering if you tested fastRTC from hugging face? I haven't done that curious about speed between this vs fastrtc vs pipecat.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#147

Post by TeMPOraL »

riquito wrote: post #111
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Very cool, thanks for sharing. A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?

FWIW, wake words are a stopgap; if we want to have a Star Trek level voice interfaces, where the computer responds only when you actually meant to call it, as opposed to using the wake word as a normal word in the conversation, the computer needs to be constantly listening.

A good analogy here is to think of the computer (assistant) as another person in the room, busy with their own stuff but paying attention to the conversations happening around them, in case someone suddenly requests their assistance.

This, of course, could be handled by a more lightweight LLM running locally and listening for explicit mentions/addressing the computer/assistant, as opposed to some context-free wake words.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#148

Post by sebra »

krick wrote: post #122

Cool for a weekend project, but honestly ChatGPT is still kinda shit at dialogues. I wonder if that's the issue with technology or OpenAI's fine-tuning (and suspect the latter), but it cannot talk like normal people do: shut up if it has nothing to add of value, ask reasonable follow-up questions if user doesn't understand something or there's ambiguity in the question. Also, on topic of follow-up questions: I don't…

I added this to personal instructions to make it less annoying:

• No compliments, flattery, or emotional rapport. • Focus on clear reasoning and evidence. • Be critical of users assumptions when needed. • Ask follow-up questions only when essential for accuracy.

However, I'm kinda concerned with crippling it by adding custom prompts. It's kinda hard to know how to use AI efficiently. But the glazing and random follow-up questions feel more like a result of some A/B testing UX-research rather than improving the results of the model.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#149

Post by justlikereddit »

peturdarri wrote: post #74
joshstrange wrote: post #42

Earlier quoted context omitted.

> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning. I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multi…

You're right, this is not solvable with regular LLMs. It's not possible to mimic natural conversational rhythm with a separate LLM generating text, a separate text-to-speech generating audio, and a separate VAD determining when to respond and when to interrupt. I strongly believe you have to do everything in one model to solve this issue, to let the model decide when to speak, when to interrupt the user even. The onl…

>It's not possible to mimic natural conversational rhythm with a separate LLM generating text, a separate text-to-speech generating audio, and a separate VAD determining when to respond and when to interrupt.

If you load the system prompt with enough assumptions that it's a speech-impared subtitle transcription that follows a dialogue you might pull it off, but likely you might need to fine tune your model to play nicely with the TTS and rest of setup

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#150

Post by Dr4kn »

justlikereddit wrote: post #144
riquito wrote: post #111

Earlier quoted context omitted.

Very cool, thanks for sharing. A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?

Modify it with an ultra light LLM agent that always listens that uses a wake word to agentically call the paid API?

You could use open wake word. Which Home Assistant developed for its own Voice Assistant
Post reply on HN