Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the current best local/open speech-to-speech setup?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#31

Post by PhilippGille »

andhuman wrote: post #29

I built this recently. I used nvidia parakeet as STT, open wake word as the wake word detection, mistral ministral 14b as LLM and pocket tts for tts. Fits snugly in my 16 gb VRAM. Pocket is small and fast and has good enough voice cloning. I first used the chatterbox turbo model, which perform better and even supported some simple paralinguistic word like (chuckle) that made it more fun, but it was just a bit too big…

OP asked:

> Is anyone doing true end-to-end speech models locally (streaming audio out), or is the SOTA still “streaming ASR + LLM + streaming TTS” glued together?

Your setup is the latter, not the former.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#33

Post by schobi »

Oh... Having a local-only voice assistant would be great. Maybe someone can share the practical side of this.

Do you have the GPU running all day at 200W to scan for wake words? Or is that running on the machine you are working on anyway?

Is this running from a headset microphone (while sitting at the desk?) or more like a USB speakerphone? Is there an Alexa jailbreak / alternative firmware as a frontend and run this on a GPU hidden away?

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#34

Post by timwis »

Home Assistant have a fully local voice assistant experience that's very pluggable and customisable. I believe it uses a fast whisper model for STT and piper for TTS.

You can run it on a raspberry pi (or ideally an N100+), and for the microphone/speaker part, you can make your own or buy their off the shelf voice hardware, which works really well.

https://www.home-assistant.io/voice-pe/

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#35

Post by albert_e »

zaken wrote: post #22
mpaepper wrote: post #3

You should look into the new Nvidia model: https://research.nvidia.com/labs/adlr/personaplex/ It has dual channel input / output and a very permissible license

Oh man that space emergency example had me rolling

Ha --

and the "Customer Service - Banking" scenario claims that it demos "accent control" and the prompt gives the agent a definitely non-indian name, yet the agents sounds 100% Indian - I found that hilarious but also isn't it a bad example given they are claiming accent control as a feature?

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#36

Post by woudsma »

Johnny_Bonk wrote: post #5

Anyone using any reasonably good small speech to text os models?

I’m using whisper with superwhisper on my mac. I’ve assigned a key on my keyboard, when I press the key it starts listening and when I release it, the text gets copied to the current cursor location. It works pretty well.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#38

Post by vidarh »

ripped_britches wrote: post #19

speech to speech is not nearly as good as livekit IMO ("old school" sequence of transcribe, LLM, synthesize). depends on what you're doing of course, but this is just because the LLMs are just way smarter than the speech to speech models which are pretty much the worst (again IMO) at anything beyond basic banter. and livekit is just a framework so you can hook it up with any models in the stack. im not an expert on t…

They work for two entirely different things. The problem with these pipelines is that unless the latency is very low they simply aren't suitable replacements for Alexa etc. For that use case, low latency beats smarts.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#39

Post by nemima »

sgt wrote: post #37

While on this subject, what's the go to transcribe speech to text model (open source or proprietary, doesn't matter) if you have to support a lot of languages really well?

If propeietary/SaaS fits your use case I can reccomend Speechmatics. Has a wider range of languages than a lot of the competition: https://speechmatics.com

(Full disclosure I'm an engineer there)

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#40

Post by dvfjsdhgfv »

sgt wrote: post #37

While on this subject, what's the go to transcribe speech to text model (open source or proprietary, doesn't matter) if you have to support a lot of languages really well?

I spent a few days on similar scenario without much success (scenario where one person speaks and then their speech is translated, and I want juts the original or both).

An API call to GPT4o works quite well (it basically handles both transcription and diarization), but I wanted a local model.

Whisper is really good for 1 person speaking. With more people you get repetitions. Qwen and other open multimodal models gives subpar results.

I tried multipass approach, with the first one identifying the language and chunking and the next one the actual transcription, but this tended to miss a lot of content.

I'm going to give canary-1b-v2 a try next weekend. But it looks like in spite of enormous development in other areas, speech recognition stalled since Whisper's release (more than 3 years already?).

Post reply on HN