Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

github.com

Post reply on HN
Print view View original

151–160 of 245 posts

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#151

Post by modeless »

tomp wrote: post #122
modeless wrote: post #101

I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…

How do you get Whisper to be fast? Isn't it quite non-realtime?

Great question! Whisper processes audio in 30 second chunks. But on a fast GPU it can finish in only 100 milliseconds or so. So you can run it 10+ times per second and get around 100ms latency. Even better actually because Whisper will predict past the end of the audio sometimes.

This is an advantage of running locally. Running whisper this way is inefficient but I have a whole GPU sitting there dedicated to one user, so it's not a problem as long as it is fast enough. It wouldn't work well for a cloud service trying to optimize GPU use. But there are other ways of doing real time speech recognition that could be used there.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#153

Post by modeless »

globalnode wrote: post #126
modeless wrote: post #108

Earlier quoted context omitted.

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…

you'd have to do something along the lines of what voice comm does to combat the output feedback problem. i think it involves an fft to analyse the two signals and cancel out the feedback, im not 100% sure on the details.

I plan to change the audio input to use WebRTC, then I get echo cancellation and network transparency for free. Although dealing with WebRTC is a headache harder than doing the AI parts.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#154

Post by yencabulator »

samsepi0l121 wrote: post #147
modeless wrote: post #101

I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…

But whisper does not support input streaming, so you have to wait for the whole llm response to trigger the transcription or not?

Apparently, by running it on windows of audio very often:

https://news.ycombinator.com/item?id=38340938

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#155

Post by joshspankit »

modeless wrote: post #108
lucubratory wrote: post #104

Earlier quoted context omitted.

How hard on your end does the task of making the chatbot converse naturally look? Specifically I'm thinking about interruptions, if it's talking too long I would like to be able to start talking and interrupt it like in a normal conversation, or if I'm saying something it could quickly interject something. Once you've got the extremely high speed, theoretically faster than real time, you can start doing that stuff ri…

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…

Short-term could it be configured as push to talk?

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#156

Post by carbocation »

jasonjmcghee wrote: post #148

Out of curiosity - to folks that have had success with this... This voice cloning is... nothing like XTTSv2, let alone ElevenLabs. It doesn't seem to care about accents at all. It does pretty well with pitch and cadence, and that's about it. I've tried all kinds of different values for alpha, beta, embedding scale, diffusion steps. Anyone else have better luck? Sure it's fast and the sound quality is pretty good, but…

I had the same experience as what you described (with a lot of experimentation with alpha and beta, as well as uploading different audio clips).

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#157

Post by modeless »

joshspankit wrote: post #155
modeless wrote: post #108

Earlier quoted context omitted.

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…

Short-term could it be configured as push to talk?

Certainly, but then it has little advantage over e.g. ChatGPT voice mode. I guess running locally is an advantage but the voice and answer quality is worse. The much better latency and more natural conversation is what I like about it.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#158

Post by Sean-Der »

modeless wrote: post #153
globalnode wrote: post #126

Earlier quoted context omitted.

you'd have to do something along the lines of what voice comm does to combat the output feedback problem. i think it involves an fft to analyse the two signals and cancel out the feedback, im not 100% sure on the details.

I plan to change the audio input to use WebRTC, then I get echo cancellation and network transparency for free. Although dealing with WebRTC is a headache harder than doing the AI parts.

What do you find hard about WebRTC?

I would love to help. Would even code up a prototype if you wanted :)

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#159

Post by dsrtslnd23 »

jasonjmcghee wrote: post #148

Out of curiosity - to folks that have had success with this... This voice cloning is... nothing like XTTSv2, let alone ElevenLabs. It doesn't seem to care about accents at all. It does pretty well with pitch and cadence, and that's about it. I've tried all kinds of different values for alpha, beta, embedding scale, diffusion steps. Anyone else have better luck? Sure it's fast and the sound quality is pretty good, but…

See the conclusion remarks in the paper - they acknowledge that voice cloning is not that good (yet).

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#160

Post by modeless »

fintechie wrote: post #133
modeless wrote: post #108

Earlier quoted context omitted.

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…

> although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. Maybe you can use some sort of speaker identification to sort this out? https://github.com/openai/whisper/discussions/264

Yes, this is a good idea. Too many good ideas, not enough time!
Post reply on HN