Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

github.com

Post reply on HN
Print view View original

91–100 of 245 posts

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#91

Post by refulgentis »

kelseyfrog wrote: post #75
refulgentis wrote: post #40

Earlier quoted context omitted.

Not sure what you mean: If you mean could inpainting and out painting with image models be faster, its a "not even wrong" question, similar to asking if the United Airlines app could get faster because American Airlines did. (Yes, getting faster is an option available to ~all code) If you mean could you inpaint and outpaint text...yes, by inserting and deleting characters. If you mean could you use an existing voice…

Ignore the speed comment; it is unrelated to my question. What I mean is, can output be conditioned on antecedent audio as well as text analogous to how image diffusion models can condition inpainting and outpatient on static parts of an image and clip embeddings?

Yes, the paper and Eleven Labs have a major feature of "given $AUDIO_SET, generate speech for $TEXT in the same style of $AUDIO_SET"

No, in that, you can't cut it at an arbitrary midword point, say at "what tim" in "what time is it bejing", and give it the string "what time is it in beijing", and have it recover seamlessly.

Yes, in that, you can cut it at an arbirtrary phoneme boundary, say 'this, I.S. a; good: test! ok?' in IPA is 'ðˈɪs, ˌaɪˌɛsˈeɪ; ɡˈʊd: tˈɛst! ˌoʊkˈeɪ?', and I can cut it 'between' a phoneme, give it the and have it complete.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#92

Post by refulgentis »

pedrovhb wrote: post #47
refulgentis wrote: post #40

Earlier quoted context omitted.

Not sure what you mean: If you mean could inpainting and out painting with image models be faster, its a "not even wrong" question, similar to asking if the United Airlines app could get faster because American Airlines did. (Yes, getting faster is an option available to ~all code) If you mean could you inpaint and outpaint text...yes, by inserting and deleting characters. If you mean could you use an existing voice…

I'm not sure I understand what you mean to say. To me it's a reasonable question asking whether text to speech models can complete a missing part of some existing speech audio, or make it go on for longer, rather than only generating speech from scratch. I don't see a connection to your faster apps analogy. Fwiw, I imagine this is possible, at least to some extent. I was recently playing with xtts and it can generate…

It imposes a cost on others when when you makes false claims like I said or felt the question was unreasonable.

I didn't and don't.

It is a hard question to understand and an interesting mind-bender to answer.

Less policing of the metacontext and more focusing on the discussion at hand will help ensure there's interlocutors around to, at the very least, continue policing.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#93

Post by ronsor »

gpm wrote: post #9
progbits wrote: post #4

> MIT license > Before using these models, you agree to [...] No, this is not MIT. If you don't like MIT license then feel free to use something else, but you can't pretend this is open source and then attempt to slap on additional restrictions on how the code can be used.

As I understand it the source code is licensed MIT, the weights are licensed "weird proprietary license that doesn't explicitly grant you any rights and implicitly probably grants you some usage rights so long as you tell the listeners or have permission from the voice you cloned". Which, if you think the weights are copyright-able in the first place, makes them practically unusable for anything commercial/that you m…

And if you don't think weights are copyrightable, it means nothing at all.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#95

Post by lhl »

eurekin wrote: post #57
stevenhuang wrote: post #41

I really want to try this but making the venv to install all the torch dependencies is starting to get old lol. How are other people dealing with this? Is there an easy way to get multiple venvs to share like a common torch venv? I can do this manually but I'm wondering if there's a tool out there that does this.

Same here. I'm using conda and eyeing simply installing a pytorch into the base conda env

I don't think "base" works like that (while it can be a fallback for some dependencies, afaik, Python packages are isolated/not in path). But even if you could, don't do it. Different packages usually have different pytorch dependencies (often CUDA as well) and it will definitely bite you.

The biggest optimization I've found is to use mamba for everything. It's ridiculously faster than conda for package resolution. With everything cached, you're mostly just waiting for your SSD at that point.

(I suppose you could add the base env's lib path to the end of your PYTHONPATH, but that sounds like a sure way to get bitten by weird dependency/reproducibility issues down the line.)

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#96

Post by eigenvalue »

eigenvalue wrote: post #50

Was somewhat annoying to get everything to work as the documentation is a bit spotty, but after ~20 minutes it's all working well for me on WSL Ubuntu 22.04. Sound quality is very good, much better than other open source TTS projects I've seen. It's also SUPER fast (at least using a 4090 GPU). Not sure it's quite up to Eleven Labs quality. But to me, what makes Eleven so cool is that they have a large library of high…

To save people some time, this is tested on Ubuntu 22.04 (google is being annoying about the download link, saying too many people have downloaded it in the past 24 hours, but if you wait a bit it should work again):

  git clone https://github.com/yl4579/StyleTTS2.git
  cd StyleTTS2
  python3 -m venv venv
  source venv/bin/activate
  python3 -m pip install --upgrade pip
  python3 -m pip install wheel
  pip install -r requirements.txt
  pip install phonemizer
  sudo apt-get install -y espeak-ng
  pip install gdown
  gdown https://drive.google.com/uc?id=1K3jt1JEbtohBLUA0X75KLw36TW7U1yxq
  7z x Models.zip
  rm Models.zip
  gdown https://drive.google.com/uc?id=1jK_VV3TnGM9dkrIMsdQ_upov8FrIymr7
  7z x Models.zip
  rm Models.zip
  pip install ipykernel pickleshare nltk SoundFile
  python -c "import nltk; nltk.download('punkt')"
  pip install --upgrade jupyter ipywidgets librosa
  python -m ipykernel install --user --name=venv --display-name="Python (venv)"
  jupyter notebook
Then navigate to /Demo and open either `Inference_LJSpeech.ipynb` or `Inference_LibriTTS.ipynb` and they should work.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#97

Post by modeless »

gjm11 wrote: post #31

HN title at present is "StyleTTS2 – open-source Eleven Labs quality Text To Speech". Actual title at the far end doesn't name any particular other product; arXiv paper linked from there doesn't mention Eleven Labs either. I thought this sort of editorializing was frowned on.

It is editorializing and it is an exaggeration. However I've been using StyleTTS2 myself and IMO it is the best open source TTS by far and definitely deserves a spot on the top of HN for a while.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#98

Post by modeless »

e12e wrote: post #80
victorbjorklund wrote: post #45

This only works for English voices right?

No? From the readme: In Utils folder, there are three pre-trained models: ASR folder: It contains the pre-trained text aligner, which was pre-trained on English (LibriTTS), Japanese (JVS), and Chinese (AiShell) corpus. It works well for most other languages without fine-tuning, but you can always train your own text aligner with the code here: yl4579/AuxiliaryASR. JDC folder: It contains the pre-trained pitch extract…

Those are just parts of the system and don't make a complete TTS. In theory you could train a complete StyleTTS2 for other languages but currently the pretrained models are English only.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#99

Post by thot_experiment »

eigenvalue wrote: post #68
eigenvalue wrote: post #63

Earlier quoted context omitted.

Ah ok, thanks. I tried the other demo.

I tried it. Sounds absolutely nothing like my voice or my wife's voice. I used the same sample files as I used 2 days ago on the Eleven Labs website, and they worked flawlessly there. So this is very, very far from being close to "Eleven Labs quality" when it comes to voice cloning.

Ah that's disappointing, have you tried https://git.ecker.tech/mrq/ai-voice-cloning ? I've had decent results with that, but inference is quite slow.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#100

Post by jsjmch »

eigenvalue wrote: post #68
eigenvalue wrote: post #63

Earlier quoted context omitted.

Ah ok, thanks. I tried the other demo.

I tried it. Sounds absolutely nothing like my voice or my wife's voice. I used the same sample files as I used 2 days ago on the Eleven Labs website, and they worked flawlessly there. So this is very, very far from being close to "Eleven Labs quality" when it comes to voice cloning.

ElevenLabs are based on Tortoise-TTS which was already pre-trained on millions of hours of data, but this one was only trained on LibriTTS which was 500 hours at best. If you have seen millions of voices, there are definitely gonna be some of them that sound like you. It is just a matter of training data, but it is very difficult to have someone collect these large amounts of data and train on it.
Post reply on HN