Viewing profile — koljab

Didn't find many coqui finetunes too so far. I have David Attenborough and Snoop Dogg finetunes on my huggingface, quality is medium. Coqui can to 17 languages. The problem with Re…

Yes, you're absolutely right. I'll provide UV and conda support soon, especially for Windows. I'm using python 3.10 still, maybe that's the issue. You can always mail me your curre…

That would be absolutely awesome. But I doubt it, since they released a shitty version of that amazing thing they put online. I feel they aren't planning to give us their top model…

Maybe possible, I did not look into that much for Coqui XTTS. What i know is that the quantized versions for Orpheus sound noticably worse. I feel audio models are quite sensitive …

LLM and TTS latency get's determined and logged at the start. It's around 220ms for the LLM returning the first synthesizable sentence fragment (depending on the length of the frag…

Lasinya voice is a XTTS 2.0.2 finetune I made with a self-created, synthesized dataset. I used https://github.com/daswer123/xtts-finetune-webui for training.

With the current 24b LLM model it's 24 GB. I have no clue how far down you can go with the GPU is using smaller models, you can set the model in server.py. Quite sure 16 GB will wo…

Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here: https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code…

Did not look into that one. Looks quite good, I will try that soon.

Yes, I tested it. I'm not that sure what they created there. It adds some noticable latency compared towards using raw websockets. Imho it's not supposed to, but it did it neverthe…

That would be quite easy to integrate. RealtimeSTT already has wakeword support for both pvporcupine and openwakewords.

It's not aware. The information that it had been interrupted would be something we can easily add to the next user chat request. Where exactly is harder, because at least for Coqui…

It's in fact using Silero via RealtimeSTT. RealtimeSTT tells when silence starts. Then a binary sentence classification model is used on the realtime transcription text which infer…

This character prompt has undergone so many iterations with LLMs it's not funny anymore. "Make her act more bold." - "She again talked about her character description, prevent that…

I doubt TTS will be fast enough for realtime without a Nvidia GPU

Dia is too slow, I need a time to first audio chunk of ~100 milliseconds. Also generations fail too often (artifacts etc)

All local models: - VAD: Webrtcvad (first fast check) followed by SileroVAD (high compute verification) - Transcription: base.en whisper (CTranslate2) - Turn Detection: KoljaB/Sent…

Thanks a lot, great insights. Exactly the kind of feedback that I need to improve things further.

Actually good question. I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model h…

That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming rea…

Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.

This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality…

I only tested it on my 4090 so far

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it insta…

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice c…