Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

github.com

Post reply on HN
Print view View original

181–190 of 245 posts

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#181

Post by rahimnathwani »

lhl wrote: post #24

I tested StyleTTS2 last month, my step-by-step notes that might be useful for people doing local setup (not too hard): https://llm-tracker.info/books/howto-guides/page/styletts-2 Also I did a little speed/quality shootoff with the LJSpeech model (vs VITS and XTTS). StyleTTS2 was pretty good and very fast: https://fediverse.randomfoo.net/notice/AaOgprU715gcT5GrZ2

Thanks. Following the instructions now. BTW mamba is no longer recommended (for those like me who aren't already using it), and the #mambaforge anchor in the link didn't work.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#182

Post by zsoltkacsandi »

GaggiX wrote: post #144
zsoltkacsandi wrote: post #130

Earlier quoted context omitted.

I was able to get it work with libjemalloc.

How fast is it on your raspberry?

Super slow. On my Mac Mini the inference was running in seconds, on Raspberry, minutes.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#183

Post by woodson »

fintechie wrote: post #133
modeless wrote: post #108

Earlier quoted context omitted.

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…

> although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. Maybe you can use some sort of speaker identification to sort this out? https://github.com/openai/whisper/discussions/264

A simple correlation of audio chunks from microphone and from the TTS should be enough to tell which parts in the input stream are re-recorded TTS. Much simpler, no?

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#184

Post by shon »

modeless wrote: post #101

I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…

Cool work! I tested it and got some mixed results:

1) it throws an error if it's installed to any drive other than C:\ --I moved it to C: and it works fine.

2) I'm seeing huge latency on an EVGA 3080Ti with 12GB. Also seeing it repeat the parsed input, even though I only spoke once, it appears to process the same input many times with slightly different predictions sometimes. Here's some logs:

Latency to LLM response: 4.59 latency to speaking: 5.31 speaking 4: Hi Jim! user spoke: Hi Jim. user spoke recently, prompting LLM. last word time: 77.81 time: 78.11742429999867 latency to prompting: 0.31

Latency to LLM response: 2.09 latency to speaking: 3.83 speaking 5: So what have you been up to lately? user spoke: So what have you been up to lately? user spoke recently, prompting LLM. last word time: 83.9 time: 84.09415280001122 latency to prompting: 0.19 user spoke: So what have you been up to lately? No, I'm watching. user spoke a while ago, ignoring. last word time: 86.9 time: 88.92142140000942 user spoke: So what have you been up to lately? No, just watching TV. user spoke a while ago, ignoring. last word time: 87.9 time: 90.76665070001036 user spoke: So what have you been up to lately? No, I'm just watching TV. user spoke a while ago, ignoring. last word time: 87.9 time: 94.16581820001011 user spoke: So what have you been up to lately? No, I'm just watching TV. user spoke a while ago, ignoring. last word time: 88.9 time: 97.85854300000938 user spoke: So what have you been up to lately? No, I'm just watching TV. user spoke a while ago, ignoring. last word time: 87.9 time: 101.54986060000374 user spoke: No, I just bought you a TV. user spoke a while ago, ignoring. last word time: 87.8 time: 104.51332219998585 user spoke: No, I'll just watch you TV. user spoke a while ago, ignoring. last word time: 87.41 time: 106.60086529998807 Latency to LLM response: 46.09 latency to speaking: 50.49

Thanks for posting it!

Edit:

3) It's hearing itself and responding to itself...

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#185

Post by yreg »

stevenhuang wrote: post #37
gjm11 wrote: post #31

HN title at present is "StyleTTS2 – open-source Eleven Labs quality Text To Speech". Actual title at the far end doesn't name any particular other product; arXiv paper linked from there doesn't mention Eleven Labs either. I thought this sort of editorializing was frowned on.

Eleven Labs is the gold standard for voice synthesis. There is nothing better out there. So it is extremely notable for an open source system to be able to approach this level of quality, which is why I'd imagine most would appreciate the comparison. I know it caught my attention.

But is this even approaching Eleven? Doesn't seem like it from the other comments here.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#186

Post by qingcharles »

satvikpendem wrote: post #23

Funnily enough, the TTS2 examples sound better than the ground truth [0]. For example, the "Then leaving the corpse within the house [...]" example has the ground truth pronounce "house" weirdly, with some change in the tonality that sounds higher, but the TTS2 version sounds more natural. I'm excited to use this for all my ePub files, many of which don't have corresponding audiobooks, such as a lot of Japanese light…

First Wife is a professional voice-over actor. I saw someone left her a bad review saying "Clearly an AI."

2023. There is no way to win.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#187

Post by modeless »

woodson wrote: post #183
fintechie wrote: post #133

Earlier quoted context omitted.

> although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. Maybe you can use some sort of speaker identification to sort this out? https://github.com/openai/whisper/discussions/264

A simple correlation of audio chunks from microphone and from the TTS should be enough to tell which parts in the input stream are re-recorded TTS. Much simpler, no?

It's not so simple when the impulse response of the room and mic and speakers are all unknown, possibly changing, plus unknown background sounds as well, possibly at a very high level. and there's also unknown latency which can be quite large especially in the networked case, and maybe some codecs, and maybe some audio "enhancement" software the OEM installed on the user's machine. Also, ideally the computer would be able to hear the user even while it is speaking.

Echo cancellation is non-trivial for sure.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#188

Post by nikkwong »

washadjeffmad wrote: post #127
nikkwong wrote: post #88

Earlier quoted context omitted.

What you're not considering here is that a large majority of this industry is made up of no-name voice actors who have a pleasant (but perfectly substitutible) voice which is now something that AI can do perfectly and at a fraction of the price. Sure, celebrities and other well-known figures will have more to gain here as they can license out their voice; but the majority of voice actors won't be able to capitalize o…

No, I am. I work with them, and I've been one (am one, rarely). I listed just one possible use, but I also see voice cloning and advanced TTS expanding access for evocative instruction, as an aid to study style and expand range. Don't be afraid on their behalf. The dooming you're talking about applied to every one of the technological changes I already listed, and we employ more performers and artists today than ever…

I don't know, I feel like the work produced through voice acting is more of a commodity than work in the other industries that you're describing. Sure, a voice actor can add a lot of emotion and verbal nuance in a way that is differentiating, but I'm not sure if the difference is enough to matter for most people for the vast majority of cases. (Or I may be too dense to realize it). This is in contradiction to say performing arts, where there are in my opinion many more dimensions to the creative output which makes it less perfectly substitutable.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#189

Post by degobah »

eigenvalue wrote: post #96
eigenvalue wrote: post #50

Was somewhat annoying to get everything to work as the documentation is a bit spotty, but after ~20 minutes it's all working well for me on WSL Ubuntu 22.04. Sound quality is very good, much better than other open source TTS projects I've seen. It's also SUPER fast (at least using a 4090 GPU). Not sure it's quite up to Eleven Labs quality. But to me, what makes Eleven so cool is that they have a large library of high…

To save people some time, this is tested on Ubuntu 22.04 (google is being annoying about the download link, saying too many people have downloaded it in the past 24 hours, but if you wait a bit it should work again): git clone https://github.com/yl4579/StyleTTS2.git cd StyleTTS2 python3 -m venv venv source venv/bin/activate python3 -m pip install --upgrade pip python3 -m pip install wheel pip install -r requirements.…

Very helpful, thanks!

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#190

Post by modeless »

shon wrote: post #184
modeless wrote: post #101

I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…

Cool work! I tested it and got some mixed results: 1) it throws an error if it's installed to any drive other than C:\ --I moved it to C: and it works fine. 2) I'm seeing huge latency on an EVGA 3080Ti with 12GB. Also seeing it repeat the parsed input, even though I only spoke once, it appears to process the same input many times with slightly different predictions sometimes. Here's some logs: Latency to LLM response…

Thanks for trying it and thanks for the feedback! Yes, right now you need to use headphones so it doesn't hear itself. Sometimes Whisper inexplicably fails to recognize speech promptly. It seems to depend on what you say, so try saying something else. I have improvements that I haven't had time to release yet that should improve the situation, and a lot more work is definitely needed, this is definitely MVP level stuff right now. This stuff is fixable but it'll take time.
Post reply on HN