Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the current best local/open speech-to-speech setup?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#41

Post by stavros »

timwis wrote: post #34

Home Assistant have a fully local voice assistant experience that's very pluggable and customisable. I believe it uses a fast whisper model for STT and piper for TTS. You can run it on a raspberry pi (or ideally an N100+), and for the microphone/speaker part, you can make your own or buy their off the shelf voice hardware, which works really well. https://www.home-assistant.io/voice-pe/

Unfortunately I didn't manage to figure out how to make their hardware to work without a HA installation. I'd really love to do that, if anyone has any info on how their protocol works, please do tell.

I looked at their Wyoming docs online but couldn't really see how to even let it find the server, and the ESPhome firmware it runs offered similarly few hints.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#42

Post by hnlmorg »

zaken wrote: post #22
mpaepper wrote: post #3

You should look into the new Nvidia model: https://research.nvidia.com/labs/adlr/personaplex/ It has dual channel input / output and a very permissible license

Oh man that space emergency example had me rolling

Oh wow. Thats definitely something…

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#44

Post by nsbk »

I'm putting together a streaming ASR + LLM + streaming TTS setup based on Nvidia speech models: nemotron ASR and magpie TTS, pipecat to glue everything together, plus an LLM of your choice. I added Spanish support using canary models, as magpie models are English-only and it still works really well.

The work is based on a repo by pipecat that I forked and modified to be more comfortable to run (docker compose for the server and client), added Spanish support via canary models, and added Nvidia Ampere support so it can run on my 3090.

The use case is a conversation partner for my gf who is learning Spanish, and it works incredibly well. For LLM I settled with Mistral-Small-3.2-24B-Instruct-2506-Q4_K_S.gguf

https://github.com/nsbk/nemotron-january-2026

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#45

Post by butvacuum »

schobi wrote: post #33

Oh... Having a local-only voice assistant would be great. Maybe someone can share the practical side of this. Do you have the GPU running all day at 200W to scan for wake words? Or is that running on the machine you are working on anyway? Is this running from a headset microphone (while sitting at the desk?) or more like a USB speakerphone? Is there an Alexa jailbreak / alternative firmware as a frontend and run this…

Wake words are generally processed extremely early in the pipeline. So if you capture audio with, say, an ESP32 the uC does the wale word watching.

Theres even microphone ADCs and DSPs(if you use a mic that outputs PCM/i2S instead of analog) that do the processing internally.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#46

Post by mikkupikku »

albert_e wrote: post #35
zaken wrote: post #22

Earlier quoted context omitted.

Oh man that space emergency example had me rolling

Ha -- and the "Customer Service - Banking" scenario claims that it demos "accent control" and the prompt gives the agent a definitely non-indian name, yet the agents sounds 100% Indian - I found that hilarious but also isn't it a bad example given they are claiming accent control as a feature?

"Sanni Virtanen", I guess it was meant to be Finnish? Maybe the "bank customer support" part threw the AI off, lmao.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#47

Post by sgt »

nemima wrote: post #39
sgt wrote: post #37

While on this subject, what's the go to transcribe speech to text model (open source or proprietary, doesn't matter) if you have to support a lot of languages really well?

If propeietary/SaaS fits your use case I can reccomend Speechmatics. Has a wider range of languages than a lot of the competition: https://speechmatics.com (Full disclosure I'm an engineer there)

Will it work with say - someone speaking English with some hindi mixed in? I'm not from there so I'm not sure how prevalent that is, but I've been told it's quite common to "mix it up" in India, and I need to probably cater for that use case.

PS if you can share your email I'll pop you an email about Speechmatics. I tried the English version and it's impressive.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#48

Post by d4rkp4ttern »

This is not strictly speech-to-speech, but I quite like it when working with Claude Code or other CLI Agents:

STT: Handy [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps confirm understanding as well as likely helps the CLI agent stay on track.

TTS: Pocket-TTS [2], just 100M params, and amazing speech quality (English only). I made a voice plugin [3] based on this, for Claude Code so it can speak out short updates whenever CC stops. It uses a non-blocking stop hook that calls a headless agent to create the 1/2-sentence summary. Turns out to be surprisingly useful. It's also fun as you can customize the speaking style and mirror your vibe etc.

The voice plugin gives commands to control it:

    /voice:speak stop
    /voice:speak azelma (change the voice)
    /voice:speak
[1] Handy https://github.com/cjpais/Handy

[2] Pocket-TTS https://github.com/kyutai-labs/pocket-tts

[3] Voice plugin for Claude Code: https://github.com/pchalasani/claude-code-tools?tab=readme-o...

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#49

Post by d4rkp4ttern »

Johnny_Bonk wrote: post #5

Anyone using any reasonably good small speech to text os models?

Parakeet V3 is near-instant transcription, and the slight accuracy drop relative to the slower/bigger Whisper models is immaterial when talking to AIs that can “read between the lines”.

Re: Ask HN: What's the current best local/open speech-to-speech setup?

#50

Post by nemima »

sgt wrote: post #47
nemima wrote: post #39

Earlier quoted context omitted.

If propeietary/SaaS fits your use case I can reccomend Speechmatics. Has a wider range of languages than a lot of the competition: https://speechmatics.com (Full disclosure I'm an engineer there)

Will it work with say - someone speaking English with some hindi mixed in? I'm not from there so I'm not sure how prevalent that is, but I've been told it's quite common to "mix it up" in India, and I need to probably cater for that use case. PS if you can share your email I'll pop you an email about Speechmatics. I tried the English version and it's impressive.

This is definitely the sort of use case we aim to support! I would need to check about Hindi specifically, but we have several bilingual models already with more to come:

https://docs.speechmatics.com/speech-to-text/languages#trans...

Drop me an email at mattn@speechmatics.com and we can chat about further details :)

Post reply on HN