I had this need recently and didn't want to use resource hungry models... so I created this:
Ask HN: What's the current best local/open speech-to-speech setup?
61–70 of 78 posts
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#62It requires a bit of tinkering, but I think pipecat is the way to go. You can plug in pretty much any STT/LLM/TTS you want and go. It definitely supports local models but its up to you to get your hands on those models. Not sure if there's any turnkey setups that are preconfigured for local install where you can just press play and go though. Last I heard E2E speech to speech models are still pretty weak. I've had pr…
I suspect the glued pipeline is going to remain dominant for a while, mostly because the intermediate text layer is structural, not just a byproduct. If you drop the text for a pure E2E model, you suddenly lose the ability to easily inject RAG context or handle complex tool use. I've been building some agent workflows recently and having that text state to pass into something like LangGraph is the only way to reliabl…
Yep, this is something end tl end models need to solve to be ideal I think. I hve seen a split brain architecture with one speaking and one thinking brain. If the thinking one could have some text tokens as output and input, to be able to refine on reasoning and rag+tools and the audio brain doing parallel audio decode.
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#63This is not strictly speech-to-speech, but I quite like it when working with Claude Code or other CLI Agents: STT: Handy [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps con…
Hi, so I'm looking for an stt that can happen on a server/cron, that will use a small local model (I have 4 vCPU threadripper CPU only and 20G ram on the server) and be able to transcribe from remote audio URLs (preferably, but I know that local models probably don't have this feature so will have to do something like curl the audio down to memory or /tmp and then transcribe and then remove the file etc). Have any th…
I’ve no thoughts on that unfortunately.
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#64This is not strictly speech-to-speech, but I quite like it when working with Claude Code or other CLI Agents: STT: Handy [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps con…
posts like this are why i visit HN daily!!! thanks for sharing your knowledge; can’t wait to try out your voice plugin
Same!
Feel free to file a gh issue if you have problems with the voice plugin
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#65Earlier quoted context omitted.
Hi, so I'm looking for an stt that can happen on a server/cron, that will use a small local model (I have 4 vCPU threadripper CPU only and 20G ram on the server) and be able to transcribe from remote audio URLs (preferably, but I know that local models probably don't have this feature so will have to do something like curl the audio down to memory or /tmp and then transcribe and then remove the file etc). Have any th…
I’ve no thoughts on that unfortunately.
:)
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#66[deleted]
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#67I'm putting together a streaming ASR + LLM + streaming TTS setup based on Nvidia speech models: nemotron ASR and magpie TTS, pipecat to glue everything together, plus an LLM of your choice. I added Spanish support using canary models, as magpie models are English-only and it still works really well. The work is based on a repo by pipecat that I forked and modified to be more comfortable to run (docker compose for the…
I got the models all the way around. Nemotron-speech ASR is the one that is English-only. Magpie TTS is multilingual and can do both English and Spanish
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#68[dead]
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#69This is not strictly speech-to-speech, but I quite like it when working with Claude Code or other CLI Agents: STT: Handy [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps con…
A 25MB TTS model: https://github.com/kittenml/kittentts
Re: Ask HN: What's the current best local/open speech-to-speech setup?
#70This is not strictly speech-to-speech, but I quite like it when working with Claude Code or other CLI Agents: STT: Handy [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps con…
A 25MB TTS model: https://github.com/kittenml/kittentts
Nice, I’ll have to try it out. They should really make a uv-installable CLI tool like pocket-TTS did. People underestimate just how much more immediately usable something becomes when you can simply get something by doing “uv tool install …”