Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the current best local/open speech-to-speech setup?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#21

Post by tuananh »

benatkin wrote: post #18
DANmode wrote: post #14

https://handy.computer got good marks from a very nontechnical user in my life this week! Local, FOSS

To save a click, it's just a fancy front end for Whisper plus a weaker CPU-only model. It has a demo video that seems impressive, but the speech is careful to sound casual while having no meaningful flaws that would cause it to mess up. If you want to make a speech to speech tool, which is what this post asks about, it would make more sense to go straight to Whisper.

you can use gpu too. i have to admit the app is very easy to use and super convenient. kudos to creator

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#23

Post by joshribakoff »

benatkin wrote: post #18
DANmode wrote: post #14

https://handy.computer got good marks from a very nontechnical user in my life this week! Local, FOSS

To save a click, it's just a fancy front end for Whisper plus a weaker CPU-only model. It has a demo video that seems impressive, but the speech is careful to sound casual while having no meaningful flaws that would cause it to mess up. If you want to make a speech to speech tool, which is what this post asks about, it would make more sense to go straight to Whisper.

I use it, sponsor it, and did a small pr. One of its goals is to be the most “forkable” starting point if i recall. But yes its just voice input. It’s meaningfully better than the mac dictation for me.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#24

Post by benatkin »

tuananh wrote: post #21
benatkin wrote: post #18

Earlier quoted context omitted.

To save a click, it's just a fancy front end for Whisper plus a weaker CPU-only model. It has a demo video that seems impressive, but the speech is careful to sound casual while having no meaningful flaws that would cause it to mess up. If you want to make a speech to speech tool, which is what this post asks about, it would make more sense to go straight to Whisper.

you can use gpu too. i have to admit the app is very easy to use and super convenient. kudos to creator

Yes, and with GPU, it's Whisper, which has been mentioned elsewhere in this article's comments. I mean that handy.computer provides the other option as a fallback for those who can't or don't want to use the GPU.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#25

Post by doonielk »

I did a MLX "streaming ASR + LLM + streaming TTS" pipeline in early 2024. I haven't worked on it since then so it's dated. There are now better versions of all the models I used.

I was able to conversational latency with the ability to interrupt the pipeline on a Mac, using a variety of tricks. It's MLX, so only relevant if you have a Mac.

https://github.com/andrewgph/local_voice

For MLX speech to speech, I've seen:

The mlx-audio package has some MLX implementations of speech to speech models: https://github.com/Blaizzy/mlx-audio/tree/main

kyutai Moshi, maybe old now but has a MLX implementation of their speech to speech model: https://github.com/kyutai-labs/moshi

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#26

Post by zahlman »

What exactly do you want the pipeline to do that cares about the input being "speech", or indeed that's different from just sending mic -> speaker directly? (I can imagine a few different things, but I want to figure out if your use case sounds like mine, or what suggestions are appropriate for what tasks.)

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#28

Post by dsrtslnd23 »

dfajgljsldkjag wrote: post #13

It requires a bit of tinkering, but I think pipecat is the way to go. You can plug in pretty much any STT/LLM/TTS you want and go. It definitely supports local models but its up to you to get your hands on those models. Not sure if there's any turnkey setups that are preconfigured for local install where you can just press play and go though. Last I heard E2E speech to speech models are still pretty weak. I've had pr…

yes, I am currently playing with pipecat - both with ASR + LLM + TTS pipeline and also speech to text (ultravox) + TTS but haven't been successful with local speech to speech setups yet.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#29

Post by andhuman »

I built this recently. I used nvidia parakeet as STT, open wake word as the wake word detection, mistral ministral 14b as LLM and pocket tts for tts. Fits snugly in my 16 gb VRAM. Pocket is small and fast and has good enough voice cloning. I first used the chatterbox turbo model, which perform better and even supported some simple paralinguistic word like (chuckle) that made it more fun, but it was just a bit too big for my rig.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#30

Post by vulkoingim »

I'm using https://spokenly.app/ in local mode, which is free. Very happy with it. It supports a bunch of models, including whisper and parakeet. Right now I'm mostly using parakeet v3 on my desktop, but it tends to do a bit more errors, although it is very fast. I cycle betwen it and Distil-Whisper Large V3.5, which is a bit slower.

On iOS I'm also using the same app, with the Apple Speech model, which I found out to be better performing for me than the parakeet/whisper. One drawback for the apple model is that you need iOS/Mac 26+ - and I haven't bothered to update to Tahoe on my mac.

Both of the models work instantly for me (Mac M1, iphone 17 Pro).

Edit: Aaaand I just saw that you're looking for speech-to-speech. Oops, still sleeping.

Post reply on HN