Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

221–230 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#221

Post by wkat4242 »

peterldowns wrote: post #96
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

Yeah I really dislike the whisperiness of this voice "Lasinya". It sounds too much like an erotic phone service. I wonder if there's any alternative voice? I don't see Lasinya even mentioned in the public coqui models: https://github.com/coqui-ai/STT-models/releases . But I don't see a list of other model names I could use either.

I tried to select kokoro in the python module but it says in the logs that only coqui is available. I do have to say the coqui models sound really good, it's just the type of voice that puts me off.

The default prompt is also way too "girlfriendy" but that was easily fixed. But for the voice, I simply don't know what the other options are for this engine.

PS: Forgive my criticism of the default voice but I'm really impressed with the responsiveness of this. It really responds so fast. Thanks for making this!

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#222

Post by tmaly »

koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

What is the min VRAM needed on the GPU to run this? I did not see that on the github

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#223

Post by Grimblewald »

krferriter wrote: post #213
jhoho wrote: post #167

Earlier quoted context omitted.

Let me introduce you to the beautiful world of virtual environments. They save you the headache of getting a full installation to run, especially when using Windows. I prefer miniconda, but venv also does the job.

`uv` is great for this because its super fast, works well as a globally installed tool (similar to conda), and can also download and manage multiple versions of python for you, and which version is used by which virtualenvironment.

While my uv use is still early days, i would second this recommendation. I've found it to have the functionallity i miss from conda in venv, but faster and more reliable than conda.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#224

Post by bb88 »

diggan wrote: post #212
bb88 wrote: post #201

Earlier quoted context omitted.

There's been a movement away from requirements.txt towards pyproject.toml. And commands like "uv add" and "uv install" take most of the pain of initializing and maintaining those dependencies.

Thanks, as mentioned, I'm not really a Python programmer so don't follow along the trends... I tried to figure out why anyone would use pyproject.toml over requirements.txt, granted they're just installing typical dependencies and didn't come up with any good answer. Personally I haven't had any issues with requirements.txt, so not sure what pyproject.toml would solve. I guess I'll change when/if I hit some bump in t…

virtualenvs are great, but they're not great on their own. requirements.txt work sorta, but then any package with more than 50 requirements requires a non-trivial amount of manual labor to maintain. (Hell even 20 deps are a pain)

Astral uv and poetry both maintain the pyproject.toml for you -- and as a bonus, they maintain the virtualenv underneath.

Then for the complete python newbs, they can run 'uv sync' or 'poetry install' and they don't have to understand what a virtualenv is -- and they don't need root, and they don't have to worry about conflicts, or which virtualenv is which, etc.

So the simple case:

    mkdir test
    cd test

    # init a new project with python 3.13
    uv init -p 3.13

    # Add project deps
    uv add numpy
    uv add ...

    # Delete the venv
    rm -rf .venv

    # reinstall everything (with the exact versions)
    uv sync

    # Install a test package in your venv
    uv pip install poetry

    # force the virtualenv back into a sane state (removing poetry and all it's deps)
    uv sync

    # update all deps
    rm uv.lock
    uv lock
Now cat your pyproject.toml, and you'll see something like this:

    [project]
    name = "test"
    version = "0.1.0"
    description = "Add your description here"
    readme = "README.md"
    requires-python = ">=3.13"
    dependencies = [
        "numpy>=2.2.5",
        "pillow>=11.2.1",
        "scipy>=1.15.2",
    ]

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#225

Post by joshspankit »

LZ_Khan wrote: post #21
smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

Honestly I think this is a problem of over-engineering and simply allowing the user to press a button when he wants to start talking and press it when he's done is good enough. Or even a codeword for start and finish. We don't need to feel like we're talking to a real person yet.

My ideal would be a small "stick remote" with a mic button.

The AI listens as long as you hold the button, and the device is efficient enough to carry with you 24/7.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#226

Post by atoav »

diggan wrote: post #212
bb88 wrote: post #201

Earlier quoted context omitted.

There's been a movement away from requirements.txt towards pyproject.toml. And commands like "uv add" and "uv install" take most of the pain of initializing and maintaining those dependencies.

Thanks, as mentioned, I'm not really a Python programmer so don't follow along the trends... I tried to figure out why anyone would use pyproject.toml over requirements.txt, granted they're just installing typical dependencies and didn't come up with any good answer. Personally I haven't had any issues with requirements.txt, so not sure what pyproject.toml would solve. I guess I'll change when/if I hit some bump in t…

There are some advantages to pyproject.toml

1. You can differenciate between different dependency groups like build dependencies, dev dependencies, test dependencies and regular dependencies. So if someone uses some dependency only in dev previously you either had to install that manually or your requirements.txt installed it for you without you needing it.

2. It adds a common description for project metadata that can be used

3. Adds a place where tool settings like those of a linter or a formatter can be stored (e.g. ruff and black)

4. Its format is standardized and allows it to be integrated with multiple build tools, toml is a bit more standardized than whatever custom file syntax python used before

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#227

Post by koljab »

wkat4242 wrote: post #221
peterldowns wrote: post #96

Earlier quoted context omitted.

Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

Yeah I really dislike the whisperiness of this voice "Lasinya". It sounds too much like an erotic phone service. I wonder if there's any alternative voice? I don't see Lasinya even mentioned in the public coqui models: https://github.com/coqui-ai/STT-models/releases . But I don't see a list of other model names I could use either. I tried to select kokoro in the python module but it says in the logs that only coqui i…

Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here:

https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

Create a subfolder in the app container: ./models/some_folder_name Copy the files from your desired voice into that folder: config.json, model.pth, vocab.json and speakers_xtts.pth (you can copy the speakers_xtts.pth from Lasinya, it's the same for every voice)

Then change the specific_model="Lasinya" line in audio_module.py into specific_model="some_folder_name".

If you change TTS_START_ENGINE to "kokoro" in server.py it's supposed to work, what does happen then? Can you post the log message?

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#228

Post by koljab »

tmaly wrote: post #222
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

What is the min VRAM needed on the GPU to run this? I did not see that on the github

With the current 24b LLM model it's 24 GB. I have no clue how far down you can go with the GPU is using smaller models, you can set the model in server.py. Quite sure 16 GB will work but at some point it will probably fail.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#229

Post by koljab »

peterldowns wrote: post #96
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

Lasinya voice is a XTTS 2.0.2 finetune I made with a self-created, synthesized dataset. I used https://github.com/daswer123/xtts-finetune-webui for training.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#230

Post by koljab »

karimf wrote: post #108
koljab wrote: post #2

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and Realtime…

Do you have any information on how long each step take? Like how many ms on each step of the pipeline? I'm curious how fast it will run if we can get this running on a Mac. Any ballpark guess?

LLM and TTS latency get's determined and logged at the start. It's around 220ms for the LLM returning the first synthesizable sentence fragment (depending on the length of the fragment, which is usually something between 3 and 10 words). Then around 80ms of TTS until the first audio chunk is delivered. STT with base.en you can neglect, it's under 5 ms, VAD same. Turn detection model also adds around 20 ms. I have zero clue if and how fast this runs on a Mac.
Post reply on HN