Ask HN: What's the current best local/open speech-to-speech setup?

#1

Post by dsrtslnd23 » Fri, Jan 23, 2026, 11:04 AM UTC

I’m trying to do the “voice assistant” thing fully locally: mic → model → speaker, low latency, ideally streaming + interruptible (barge-in).

Qwen3 Omni looks perfect on paper (“real-time”, speech-to-speech, etc). But I’ve been poking around and I can’t find a single reproducible “here’s how I got the open weights doing real speech-to-speech locally” writeup. Lots of “speech in → text out” or “audio out after the model finishes”, but not a usable realtime voice loop. Feels like either (a) the tooling isn’t there yet, or (b) I’m missing the secret sauce.

What are people actually using in 2026 if they want open + local voice?

Is anyone doing true end-to-end speech models locally (streaming audio out), or is the SOTA still “streaming ASR + LLM + streaming TTS” glued together?

If you did get Qwen3 Omni speech-to-speech working: what stack (transformers / vLLM-omni / something else), what hardware, and is it actually realtime?

What’s the most “works today” combo on a single GPU?

Bonus: rough numbers people see for mic → first audio back

Would love pointers to repos, configs, or “this is the one that finally worked for me” war stories.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#2

Post by jauntywundrkind » Fri, Jan 23, 2026, 9:39 PM UTC

It was a little annoying getting old qt5 tools installed but I really enjoyed using dsnote / Speech Note. Huge model selection for my amd gpu. Good tool. I haven't done enough specific studying yet to give you suggestions for which model to go with. WhisperFlow is very popular.

Kyutai some very interesting work always. Their delayed streams work is bleeding edge & sounds very promising especially for low latency. Not sure why I have not yet tried it tbh. https://github.com/kyutai-labs/delayed-streams-modeling

There's also a really nice elegant simple app Handy. Only supports Whisper and Parakeet V3 but nice app & those are amazing models. https://github.com/cjpais/Handy

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#3

Post by mpaepper » Fri, Jan 23, 2026, 9:50 PM UTC

You should look into the new Nvidia model: https://research.nvidia.com/labs/adlr/personaplex/

It has dual channel input / output and a very permissible license

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#4

Post by dsrtslnd23 » Fri, Jan 23, 2026, 9:58 PM UTC

mpaepper wrote: post #3

You should look into the new Nvidia model: https://research.nvidia.com/labs/adlr/personaplex/ It has dual channel input / output and a very permissible license

oh - very interesting indeed! thanks

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#5

Post by Johnny_Bonk » Fri, Jan 23, 2026, 11:29 PM UTC

Anyone using any reasonably good small speech to text os models?

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#6

Post by garblegarble » Fri, Jan 23, 2026, 11:32 PM UTC

Johnny_Bonk wrote: post #5

Anyone using any reasonably good small speech to text os models?

For my inputs, whisper distil-large-v3.5 is the best. I tried Parakeet 0.6 v3 last night but it has higher error rates than I'd like (but it is fast...)

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#7

Post by amelius » Fri, Jan 23, 2026, 11:35 PM UTC

For the TTS part: https://github.com/supertone-inc/supertonic

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#8

Post by Johnny_Bonk » Fri, Jan 23, 2026, 11:35 PM UTC

garblegarble wrote: post #6

Johnny_Bonk wrote: post #5

Anyone using any reasonably good small speech to text os models?

For my inputs, whisper distil-large-v3.5 is the best. I tried Parakeet 0.6 v3 last night but it has higher error rates than I'd like (but it is fast...)

Nice I'll try it, as of now for my personal stt workflow I use eleven labs api which is pretty generous but curious to play around with other options

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#9

Post by garblegarble » Fri, Jan 23, 2026, 11:46 PM UTC

Johnny_Bonk wrote: post #8

garblegarble wrote: post #6

Earlier quoted context omitted.

For my inputs, whisper distil-large-v3.5 is the best. I tried Parakeet 0.6 v3 last night but it has higher error rates than I'd like (but it is fast...)

Nice I'll try it, as of now for my personal stt workflow I use eleven labs api which is pretty generous but curious to play around with other options

I assume that will be better than whisper - I haven't benchmarked it against cloud models, the project I'm working on cannot send data out to cloud models

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#10

Post by cbrews » Fri, Jan 23, 2026, 11:49 PM UTC

mpaepper wrote: post #3

You should look into the new Nvidia model: https://research.nvidia.com/labs/adlr/personaplex/ It has dual channel input / output and a very permissible license

Thanks for sharing this! I'm going to put this on my list to play around with. I'm not really an expert in this tech, I come from the audio background, but recently was playing around with streaming Speech-to-Text (using Whisper) / Text-to-Speech (using Kokoro at the time) on a local machine.

The most challenging part in my build was tuning the inference batch sizing here. I was able to get it working well for Speech-to-Text down to batch sizes of 200ms. I even implement a basic local agreement algorithm and it was still very fast (inferencing time, I think, was around 10-20ms?). You're basically limited by the minimum batch size, NOT inference time. Maybe that's a missing "secret sauce" suggested in the original post?

In the use case listed above, the TTS probably isn't a bottleneck as long as OP can generate tokens quickly.

All this being said a wrapped model like this that is able to handle hand-offs between these parts of the process sounds really useful and I'll definitely be interested in seeing how it performs.

Let me know if you guys play with this and find success.