Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

231–238 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#231

Post by koljab »

nardi wrote: post #126
koljab wrote: post #34

Earlier quoted context omitted.

Actually good question. I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model heavily on a single language but again that alone does not speed up inference. To gain speed you'd have to bring the parameter count down and train the model from scratch with a single language only. That might work but it's also quite proba…

I don't know what I'm talking about, but could you use distillation techniques?

Maybe possible, I did not look into that much for Coqui XTTS. What i know is that the quantized versions for Orpheus sound noticably worse. I feel audio models are quite sensitive to quantization.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#232

Post by koljab »

echelon wrote: post #114
koljab wrote: post #41

Earlier quoted context omitted.

All local models: - VAD: Webrtcvad (first fast check) followed by SileroVAD (high compute verification) - Transcription: base.en whisper (CTranslate2) - Turn Detection: KoljaB/SentenceFinishedClassification (selftrained BERT-model) - LLM: hf.co/bartowski/huihui-ai_Mistral-Small-24B-Instruct-2501-abliterated-GGUF:Q4_K_M (easily switchable) - TTS: Coqui XTTSv2, switchable to Kokoro or Orpheus (this one is slower)

That's excellent. Really amazing bringing all of these together like this. Hopefully we get an open weights version of Sesame [1] soon. Keep watching for it, because that'd make a killer addition to your app. [1] https://www.sesame.com/

That would be absolutely awesome. But I doubt it, since they released a shitty version of that amazing thing they put online. I feel they aren't planning to give us their top model soon.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#233

Post by koljab »

sabellito wrote: post #164

Every time I see these things, they look cool as hell, I get excited, then I try to get them working on my gaming PC (that has the GPU), I spend 1-2h fighting with python and give up. Today's issue is that my python version is 3.12 instead of =3.9. Installing python 3.11 from the official website does nothing, I give up. It's a shame that the amazing work done by people like the OP gets underused because of this mess…

Yes, you're absolutely right. I'll provide UV and conda support soon, especially for Windows. I'm using python 3.10 still, maybe that's the issue. You can always mail me your current problem or log an issue, I really care about opening up the repo for as many ppl to use as possible.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#234

Post by turnsout »

releaslot wrote: post #189
turnsout wrote: post #188

Earlier quoted context omitted.

And this works about 25% of the time. The rest of the time, there is some inscrutable error with the version number of a dependency in requirements.txt or something similar, which you end up Googling, only to find an open issue on a different project's Github repo. Someone needs to make an LLM agent that just handles Python dependency hell.

"Someone needs to make an LLM agent that just handles Python dependency hell." This is why they are constantly delaying GPT 5.

Haha fair enough—if they can solve the Python situation, I'd be fine slapping the "AGI" label on it.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#235

Post by wkat4242 »

koljab wrote: post #227
wkat4242 wrote: post #221

Earlier quoted context omitted.

Yeah I really dislike the whisperiness of this voice "Lasinya". It sounds too much like an erotic phone service. I wonder if there's any alternative voice? I don't see Lasinya even mentioned in the public coqui models: https://github.com/coqui-ai/STT-models/releases . But I don't see a list of other model names I could use either. I tried to select kokoro in the python module but it says in the logs that only coqui i…

Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here: https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a... Create a subfolder in the app container: ./models/some_folder_name Copy the files from your desired voice into that folder: config.json, model.pth, vocab.json and speakers_xtts.pth (you can copy the speakers_xtts.pth from Lasinya, it'…

Thank you!

I didn't realise that you custom-made that voice. Would you have some links to other out-of-the-box voices for coqui? I'm having some trouble finding them. I think from seeing the demo page that the idea is that you clone someone else's voice or something with that engine. Because I don't see any voices listed. I've never seen it before.

And yes I switched to Kokoro now, I thought it was the default already but then I saw there were 3 lines configuring the same thing. So that's working. Kokoro isn't quite as good though as coqui, that's why I'm wondering about that. I also used kokoro on openwebui and I wasn't very happy with it there either. It's fast, but some pronounciation is weird. Also, it would be amazing to have bilingual TTS (English/Spanish in my case). And it looks like Coqui might be able to do that.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#236

Post by koljab »

wkat4242 wrote: post #235
koljab wrote: post #227

Earlier quoted context omitted.

Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here: https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a... Create a subfolder in the app container: ./models/some_folder_name Copy the files from your desired voice into that folder: config.json, model.pth, vocab.json and speakers_xtts.pth (you can copy the speakers_xtts.pth from Lasinya, it'…

Thank you! I didn't realise that you custom-made that voice. Would you have some links to other out-of-the-box voices for coqui? I'm having some trouble finding them. I think from seeing the demo page that the idea is that you clone someone else's voice or something with that engine. Because I don't see any voices listed. I've never seen it before. And yes I switched to Kokoro now, I thought it was the default alread…

Didn't find many coqui finetunes too so far. I have David Attenborough and Snoop Dogg finetunes on my huggingface, quality is medium.

Coqui can to 17 languages. The problem with RealtimeVoiceChat repo is turn detection, the model I use to determine if a partial sentence indicates turn change is trained on english corpus only.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#238

Post by robbomacrae »

kadushka wrote: post #79
varispeed wrote: post #52

Earlier quoted context omitted.

> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

Read this if you haven’t already: https://marshallbrain.com/manna1 That’s a much more serious anxiety trigger for me.

I just wanted to say thanks for the recommendation! Really good read.
Post reply on HN