Live data from Hacker News

StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

github.com

21–30 of 245 posts

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#21
post #4

> MIT license > Before using these models, you agree to [...] No, this is not MIT. If you don't like MIT license then feel free to use something else, but you can't pretend this is open source and then attempt to slap on additional restrictions on how the code can be used.

As if anyone outside of corporate legal actually cares

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#22
post #12

We're now at "free, local, AI friend that you can have conversations with on consumer hardware" territory. - synthesize an avatar using stablediffusion - synthesize conversation with llama - synthesize the voice with this text thing soon - VR - Video wild times!

Yup, and you can already mix and match both local and cloud AIs with stuff like SillyTavern/RealmPlay if you wanna try what the experience is like, people have been using it to roleplay for a while.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#23
Funnily enough, the TTS2 examples sound better than the ground truth [0]. For example, the "Then leaving the corpse within the house [...]" example has the ground truth pronounce "house" weirdly, with some change in the tonality that sounds higher, but the TTS2 version sounds more natural.

I'm excited to use this for all my ePub files, many of which don't have corresponding audiobooks, such as a lot of Japanese light novels. I am currently using Moon+ Reader on Android which has TTS but it is very robotic.

[0] https://styletts2.github.io/

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#24
I tested StyleTTS2 last month, my step-by-step notes that might be useful for people doing local setup (not too hard): https://llm-tracker.info/books/howto-guides/page/styletts-2

Also I did a little speed/quality shootoff with the LJSpeech model (vs VITS and XTTS). StyleTTS2 was pretty good and very fast: https://fediverse.randomfoo.net/notice/AaOgprU715gcT5GrZ2

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#26
post #13
post #12

We're now at "free, local, AI friend that you can have conversations with on consumer hardware" territory. - synthesize an avatar using stablediffusion - synthesize conversation with llama - synthesize the voice with this text thing soon - VR - Video wild times!

Which consumer gpu runs llama 70B?

A Mac with a lot of unified RAM can do it, or a dual 3090/4090 setup gets you 48gb of VRAM.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#27
post #13
post #12

We're now at "free, local, AI friend that you can have conversations with on consumer hardware" territory. - synthesize an avatar using stablediffusion - synthesize conversation with llama - synthesize the voice with this text thing soon - VR - Video wild times!

Which consumer gpu runs llama 70B?

A single 3090, or any 24GB GPU. Just barely.

Yi 34B is a much better fit. I can cram 75K context onto 24GB without brutalizing the model with <3bpw quantization, like you have to do with 70B for 4K context.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#28

Why name it Style if it isn't a StyleGAN? Looks like the first one wasn't either. Interesting to see moves away from flows, especially when none of the flows were modern. Also, is no one clicking on the audio links? There are some... questionable ones... and I'm pretty sure lots of mistakes.

> Looks like the first one wasn't either.

The first one says it uses AdaIN layers to help control style? https://arxiv.org/pdf/2205.15439.pdf#page=2 Seems as justifiable as the original StyleGAN calling itself StyleX...

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#29

Why name it Style if it isn't a StyleGAN? Looks like the first one wasn't either. Interesting to see moves away from flows, especially when none of the flows were modern. Also, is no one clicking on the audio links? There are some... questionable ones... and I'm pretty sure lots of mistakes.

It's not called a GAN TTS right? StyleGAN is called what it is because of a "style-based" approach and StyleTTS/2 seems to be doing the same (applying style transfer) through different method (and disentangling style from the rest of the voice synthesis).

(Actually, looked at the original StyleTTS paper and it actually even partially uses AdaIN in the decoder, which is the same way that StyleGAN injected style information? Still, I think is besides the point for the naming.)

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#30
post #24

I tested StyleTTS2 last month, my step-by-step notes that might be useful for people doing local setup (not too hard): https://llm-tracker.info/books/howto-guides/page/styletts-2 Also I did a little speed/quality shootoff with the LJSpeech model (vs VITS and XTTS). StyleTTS2 was pretty good and very fast: https://fediverse.randomfoo.net/notice/AaOgprU715gcT5GrZ2

> inferences at up to 15-95X (!) RT on my 4090

That's incredible!

Are infill and outpainting equivalents possible? Super-RT TTS at this level of quality opens up a diverse array of uses esp for indie/experimental gamedev that I'm excited for.

Post reply on HN