Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

11–20 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#11

Post by koljab »

ivape wrote: post #3
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Would you say you are using the best-in-class speech to text libs at the moment? I feel like this space is moving fast because the last time I was headed down this track, I was sure whisper-cpp was the best.

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#13

Post by koljab »

dotancohen wrote: post #7
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

This looks great. What hardware do you use, or have you tested it on?

I only tested it on my 4090 so far

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#14

Post by ivape »

koljab wrote: post #11
ivape wrote: post #3

Earlier quoted context omitted.

Would you say you are using the best-in-class speech to text libs at the moment? I feel like this space is moving fast because the last time I was headed down this track, I was sure whisper-cpp was the best.

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.

Yeah, I figured you would know. Thanks for that, bookmarking that asr leaderboard.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#15

Post by koljab »

IshKebab wrote: post #12

Impressive! I guess the speech synthesis quality is the best available open source at the moment? The endgame of this is surely a continuously running wave to wave model with no text tokens at all? Or at least none in the main path.

This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality I'd say there are better models out there.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#16

Post by WhitneyLand »

smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

>>they don't allow talking naturally

Neither do phone calls. Round trip latency can easily be 300ms, which we’ve all learned to adapt our speech to.

If you want to feel true luxury find an old analog PTSN line. No compression artifacts or delays. Beautiful and seamless 50ms latency.

Digital was a terrible event for call quality.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#17

Post by [deleted] »

koljab wrote: post #15
IshKebab wrote: post #12

Impressive! I guess the speech synthesis quality is the best available open source at the moment? The endgame of this is surely a continuously running wave to wave model with no text tokens at all? Or at least none in the main path.

This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality I'd say there are better models out there.

[deleted]

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#18

Post by kristopolous »

koljab wrote: post #11
ivape wrote: post #3

Earlier quoted context omitted.

Would you say you are using the best-in-class speech to text libs at the moment? I feel like this space is moving fast because the last time I was headed down this track, I was sure whisper-cpp was the best.

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.

https://yummy-fir-7a4.notion.site/dia is the new hotness.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#19

Post by zaggynl »

koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Neat! I'm already using openwebui/ollama with a 7900 xtx but the STT and TTS parts don't seem to work with it yet:

2025-05-05 20:53:15,808] [WARNING] [real_accelerator.py:194:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.

Error loading model for checkpoint ./models/Lasinya: This op had not been implemented on CPU backend.

Post reply on HN