Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

21–30 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#21

Post by LZ_Khan »

smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

Honestly I think this is a problem of over-engineering and simply allowing the user to press a button when he wants to start talking and press it when he's done is good enough. Or even a codeword for start and finish.

We don't need to feel like we're talking to a real person yet.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#22

Post by koljab »

kristopolous wrote: post #18
koljab wrote: post #11

Earlier quoted context omitted.

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.

https://yummy-fir-7a4.notion.site/dia is the new hotness.

Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#24

Post by PhunkyPhil »

breaker-kind wrote: post #23

why is your AI chatbot talking in a bizarre attempt at AAVE?

This is the system prompt

https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/s...

My favorite line:

"You ARE this charming, witty, wise girlfriend. Don't explain how you're talking or thinking; just be that person."

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#25

Post by joshstrange »

This is very, very cool! The interrupting was a "wow" moment for me (I know it's not "new new" but to see it so well done in open source was awesome).

Question about the Interrupt feature, how does it handle "Mmk", "Yes", "Of course", "cough", etc? Aside from the sycophancy from OpenAI's voice chat (no, not every question I ask is a "great question!") I dislike that a noise sometimes stops the AI from responding and there isn't a great way to get back on track, to pick up where you left off.

It's a hard problem, how do you stop replying quickly AND make sure you are stopping for a good reason?

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#26

Post by [deleted] »

LZ_Khan wrote: post #21
smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

Honestly I think this is a problem of over-engineering and simply allowing the user to press a button when he wants to start talking and press it when he's done is good enough. Or even a codeword for start and finish. We don't need to feel like we're talking to a real person yet.

[deleted]

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#27

Post by joshstrange »

smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

This 100%, yes!

I've found myself putting in filler words or holding a noise "Uhhhhhhhhh" while I'm trying to form a thought but I don't want the LLM to start replying. It's a really hard problem for sure. Similar to the problem of allowing for interruptions but not stopping if the user just says "Right!", "Yes", aka active listening.

One thing I love about MacWhisper (not special to just this STT tool) is it's hold to talk so I can stop talking for as long as I want then start again without it deciding I'm done.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#28

Post by kristopolous »

koljab wrote: post #22
kristopolous wrote: post #18

Earlier quoted context omitted.

https://yummy-fir-7a4.notion.site/dia is the new hotness.

Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.

alright, dumb question.

(1) I assume these things can do multiple languages

(2) Given (1), can you strip all the languages you aren't using and speed things up?

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#29

Post by koljab »

joshstrange wrote: post #25

This is very, very cool! The interrupting was a "wow" moment for me (I know it's not "new new" but to see it so well done in open source was awesome). Question about the Interrupt feature, how does it handle "Mmk", "Yes", "Of course", " cough ", etc? Aside from the sycophancy from OpenAI's voice chat (no, not every question I ask is a "great question!") I dislike that a noise sometimes stops the AI from responding an…

That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming realtime transcription as a trigger. That adds a bit of latency but that gets compensated by way better accuracy.

Edit: just realized the irony but it's really a good question lol

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#30

Post by echelon »

koljab wrote: post #13
dotancohen wrote: post #7

Earlier quoted context omitted.

This looks great. What hardware do you use, or have you tested it on?

I only tested it on my 4090 so far

Are you using all local models, or does it also use cloud inference? Proprietary models?

Which models are running in which places?

Cool utility!

Post reply on HN