Show HN: Real-time AI Voice Chat at ~500ms Latency
1–10 of 238 posts
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#2I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs.
Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8
The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and RealtimeTTS (supporting engines like Coqui XTTSv2/Kokoro) to achieve around 500ms response latency, even when running larger local models like a 24B Mistral fine-tune via Ollama.
Key aspects: Designed for local LLMs (Ollama primarily, OpenAI connector included). Interruptible conversation. Smart turn detection to avoid cutting the user off mid-thought. Dockerized setup available for easier dependency management.
It requires a decent CUDA-enabled GPU for good performance due to the STT/TTS models.
Would love to hear your feedback on the approach, performance, potential optimizations, or any features you think are essential for a good local voice AI experience.
The code is here: https://github.com/KoljaB/RealtimeVoiceChat
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#3I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…
Would you say you are using the best-in-class speech to text libs at the moment? I feel like this space is moving fast because the last time I was headed down this track, I was sure whisper-cpp was the best.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#4Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons.
With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools.
I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what he was saying. Length of pauses wasn't a problem. I don't how complex that problem is though. Probably another AI needs to analyse the input so far a decide if it's a pause or not.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#5Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…
Maybe we should settle on some special sound or word which officially signals that we're making a pause for whatever reason, but that we intend to continue with dictating in a couple of seconds. Like "Hmm, wait".
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#6Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…
Maybe we should settle on some special sound or word which officially signals that we're making a pause for whatever reason, but that we intend to continue with dictating in a couple of seconds. Like "Hmm, wait".
Alternatively we could pretend it’s a radio and follow those conventions.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#7I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…
This looks great. What hardware do you use, or have you tested it on?
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#8Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…
Could that not work with simple instructions? Let the AI decide to respond only with a special wait token until it thinks you are ready. Might not work perfectly but would be a start.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#9Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…
Yeah, when I am trying to learn about a topic, I need to think about my question, you know, pausing mid-sentence. All the products jump in and interrupt, no matter if I tell them not to do so. Non-annoying humans don't jump in to fill the gap, they read my face, they take cues, then wait for me to finish. Its one thing to ask an AI to give me directions to the nearest taco stand, its another to have a dialogue about complex topics.
Re: Show HN: Real-time AI Voice Chat at ~500ms Latency
#10Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…
Maybe we should settle on some special sound or word which officially signals that we're making a pause for whatever reason, but that we intend to continue with dictating in a couple of seconds. Like "Hmm, wait".
Two input streams sounds like a good hacky solution. One input stream captures everything, the second is on the look out for your filler words like "um, aahh, waaiit, no nevermind, scratch that". The second stream can act as the veto-command and cut off the LLM. A third input stream can simply be on the lookout for long pauses. All this gets very resource intensive quickly. I been meaning to make this but since I haven't, I'm going to punish myself and just give the idea away. Hopefully I'll learn my lesson.