Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

github.com

Post reply on HN
Print view View original

71–80 of 245 posts

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#71

Post by eginhard »

api wrote: post #20

It should be pretty easy to make training data for TTS. The Whisper STT models are open so just chop up a ton of audio and use Whisper to annotate it, then train the other direction to produce audio from text. So you’re basically inverting Whisper.

STT training data includes all kinds of "noisy" speech so that the model learns to recognise speech in any conditions. TTS training data needs to be as clean as possible so that you don't introduce artefacts in the output and this high-quality data is much harder to get. A simple inversion is not really feasible or at least requires filtering out much of the data.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#72

Post by lucubratory »

stevenhuang wrote: post #37
gjm11 wrote: post #31

HN title at present is "StyleTTS2 – open-source Eleven Labs quality Text To Speech". Actual title at the far end doesn't name any particular other product; arXiv paper linked from there doesn't mention Eleven Labs either. I thought this sort of editorializing was frowned on.

Eleven Labs is the gold standard for voice synthesis. There is nothing better out there. So it is extremely notable for an open source system to be able to approach this level of quality, which is why I'd imagine most would appreciate the comparison. I know it caught my attention.

OpenAI's TTS is better than Eleven Labs, but they don't let you train it to have a particular voice out of fear of the consequences.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#75

Post by kelseyfrog »

refulgentis wrote: post #40
kelseyfrog wrote: post #30

Earlier quoted context omitted.

> inferences at up to 15-95X (!) RT on my 4090 That's incredible! Are infill and outpainting equivalents possible? Super-RT TTS at this level of quality opens up a diverse array of uses esp for indie/experimental gamedev that I'm excited for.

Not sure what you mean: If you mean could inpainting and out painting with image models be faster, its a "not even wrong" question, similar to asking if the United Airlines app could get faster because American Airlines did. (Yes, getting faster is an option available to ~all code) If you mean could you inpaint and outpaint text...yes, by inserting and deleting characters. If you mean could you use an existing voice…

Ignore the speed comment; it is unrelated to my question.

What I mean is, can output be conditioned on antecedent audio as well as text analogous to how image diffusion models can condition inpainting and outpatient on static parts of an image and clip embeddings?

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#76

Post by huac »

kelseyfrog wrote: post #30
lhl wrote: post #24

I tested StyleTTS2 last month, my step-by-step notes that might be useful for people doing local setup (not too hard): https://llm-tracker.info/books/howto-guides/page/styletts-2 Also I did a little speed/quality shootoff with the LJSpeech model (vs VITS and XTTS). StyleTTS2 was pretty good and very fast: https://fediverse.randomfoo.net/notice/AaOgprU715gcT5GrZ2

> inferences at up to 15-95X (!) RT on my 4090 That's incredible! Are infill and outpainting equivalents possible? Super-RT TTS at this level of quality opens up a diverse array of uses esp for indie/experimental gamedev that I'm excited for.

It is theoretically possible to train a model that, given some speech, attempts to continue the speech, e.g. Spectron: https://michelleramanovich.github.io/spectron/spectron/. Similarly, it is possible to train a model to edit the content, a la Voicebox: https://voicebox.metademolab.com/edit.html.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#78

Post by speedgoose »

brucethemoose2 wrote: post #27
jpeter wrote: post #13

Earlier quoted context omitted.

Which consumer gpu runs llama 70B?

A single 3090, or any 24GB GPU. Just barely. Yi 34B is a much better fit. I can cram 75K context onto 24GB without brutalizing the model with <3bpw quantization, like you have to do with 70B for 4K context.

Can it produce any meaningful outputs with such an extreme quantisation?

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#79

Post by huac »

lucubratory wrote: post #72
stevenhuang wrote: post #37

Earlier quoted context omitted.

Eleven Labs is the gold standard for voice synthesis. There is nothing better out there. So it is extremely notable for an open source system to be able to approach this level of quality, which is why I'd imagine most would appreciate the comparison. I know it caught my attention.

OpenAI's TTS is better than Eleven Labs, but they don't let you train it to have a particular voice out of fear of the consequences.

I concur that, for the use cases that OpenAI's voices cover, it is significantly better than Eleven.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#80

Post by e12e »

victorbjorklund wrote: post #45

This only works for English voices right?

No? From the readme:

In Utils folder, there are three pre-trained models:

    ASR folder: It contains the pre-trained text aligner, which was pre-trained on English (LibriTTS), Japanese (JVS), and Chinese (AiShell) corpus. It works well for most other languages without fine-tuning, but you can always train your own text aligner with the code here: yl4579/AuxiliaryASR.

    JDC folder: It contains the pre-trained pitch extractor, which was pre-trained on English (LibriTTS) corpus only. However, it works well for other languages too because F0 is independent of language. If you want to train on singing corpus, it is recommended to train a new pitch extractor with the code here: yl4579/PitchExtractor.

    PLBERT folder: It contains the pre-trained PL-BERT model, which was pre-trained on English (Wikipedia) corpus only. It probably does not work very well on other languages, so you will need to train a different PL-BERT for different languages using the repo here: yl4579/PL-BERT. You can also replace this module with other phoneme BERT models like XPhoneBERT which is pre-trained on more than 100 languages.
Post reply on HN