Ask HN: What's the current best local/open speech-to-speech setup? • Hacker News Board

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#11

Post by BiraIgnacio » Fri, Jan 23, 2026, 11:52 PM UTC

garblegarble wrote: post #6

Johnny_Bonk wrote: post #5

Anyone using any reasonably good small speech to text os models?

For my inputs, whisper distil-large-v3.5 is the best. I tried Parakeet 0.6 v3 last night but it has higher error rates than I'd like (but it is fast...)

oh I've been looking into whisper and vosk in the last few days. I'll probably go with whisper (with whisper.cpp) but has anyone compared it to vosk models?

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#12

Post by hackomorespacko » Sat, Jan 24, 2026, 12:16 AM UTC

[flagged]

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#13

Post by dfajgljsldkjag » Sat, Jan 24, 2026, 12:53 AM UTC

It requires a bit of tinkering, but I think pipecat is the way to go. You can plug in pretty much any STT/LLM/TTS you want and go. It definitely supports local models but its up to you to get your hands on those models.

Not sure if there's any turnkey setups that are preconfigured for local install where you can just press play and go though.

Last I heard E2E speech to speech models are still pretty weak. I've had pretty bad results from gpt-realtime and that's a proprietary model, I'm assuming open source is a bit behind.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#14

Post by DANmode » Sat, Jan 24, 2026, 1:07 AM UTC

https://handy.computer got good marks from a very nontechnical user in my life this week!

Local, FOSS

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#15

Post by marsbars241 » Sat, Jan 24, 2026, 1:49 AM UTC

Tangential: What hardware are you using for the interface on these? Is there a good array microphone that performs on par with echos/ghomes/homepods?

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#16

Post by varik77 » Sat, Jan 24, 2026, 1:52 AM UTC

I have used https://github.com/SaynaAI/sayna . What I like the most is that you can switch between the providers easily and see what works for you the best. It also supports local models.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#17

Post by hedgehog » Sat, Jan 24, 2026, 2:04 AM UTC

I haven't tried them myself but the Kyutai has a couple projects that could fit.

https://kyutai.org

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#18

Post by benatkin » Sat, Jan 24, 2026, 3:50 AM UTC

DANmode wrote: post #14

https://handy.computer got good marks from a very nontechnical user in my life this week! Local, FOSS

To save a click, it's just a fancy front end for Whisper plus a weaker CPU-only model. It has a demo video that seems impressive, but the speech is careful to sound casual while having no meaningful flaws that would cause it to mess up. If you want to make a speech to speech tool, which is what this post asks about, it would make more sense to go straight to Whisper.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#19

Post by ripped_britches » Sat, Jan 24, 2026, 4:29 AM UTC

speech to speech is not nearly as good as livekit IMO ("old school" sequence of transcribe, LLM, synthesize). depends on what you're doing of course, but this is just because the LLMs are just way smarter than the speech to speech models which are pretty much the worst (again IMO) at anything beyond basic banter. and livekit is just a framework so you can hook it up with any models in the stack. im not an expert on the local parts but i would assume this pretty easy to glue together.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#20

Post by soulofmischief » Sat, Jan 24, 2026, 4:30 AM UTC

I have a great local assistant that works end-to-end with voice. It's built on local, web-first technologies, it fits small LLMs in memory and manages inference and TTS/STT without stuttering. I've been shaping it up over a couple years and constantly switching out new models.

If you want something simple that runs in browser, look at vosk-browser[0] and vits-web[1].

I'd also recommend checking out KittenTTS[2], I use it and it's great for the size/performance. However, you'd need to implement a custom JavaScript harness for the model since it's a python project. If you need help with that, shoot me an email and I can share some code.

There are other great approaches too if you don't mind python, personally I chose the web as a platform in order to make my agent fully portable and remote once I release it.

And of course, NVIDIA's new model just came out last week[3] but I haven't gotten to test it out just yet, and also there was the recent Sparrow-1[4] announcement which shows people are finally putting money into the problems plaguing voice agents that are rigged up from several models and glue infrastructure, vs a single end-to-end model or at least a conversational turn-taking model to keep things on rails.

[0] https://www.npmjs.com/package/vosk-browser

[1] https://github.com/diffusionstudio/vits-web

[2] https://github.com/KittenML/KittenTTS

[3] https://research.nvidia.com/labs/adlr/personaplex/

[4] https://www.tavus.io/post/sparrow-1-human-level-conversation...