Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

github.com

Post reply on HN
Print view View original

221–230 of 245 posts

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#221

Post by nmstoker »

eigenvalue wrote: post #137
modeless wrote: post #101

I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…

Tried it but it seems it only works with Cuda 11 and I have 12 installed. Not really willing to potentially screw up my Cuda environment to try it.

Using a conda environment should be able to get around that I believe

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#222

Post by refulgentis »

IshKebab wrote: post #197
refulgentis wrote: post #162

Earlier quoted context omitted.

It's not, at all. He could have meant speed, text, audio, words, or phonemes, with least probably images. He probably didn't mean phonemes or he wouldn't be asking. He probably didn't mean arbitrarily slicing 'real' audio and stitching on fake audio - he made repeated references to a video game. He probably didn't mean inpainting and outpainting imagery, even though he made reference to a video game, because its an a…

Inpainting and outpainting of images is when the model generates bits inside or outside the image that don't exist. By analogy he was talking about generating sound inside (I.e. filling gaps) or outside (extrapolating beyond the end) the audio. I don't know why you would think he was talking about inpainting images, words. This whole discussion is about speech synthesis.

Right, _until he brought up inpainting and outpainting_. And as I already laid out, the audio options made just about as much sense as the art.

I honestly can't believe how committed you are to explaining to me that as the only person who bothered answering, I'm the problem.

I've been in AI art when it was 10 people in an IRC room trying to figure out what to do with a bunch of GPUs an ex-hedge fund manager snapped up, and spent the last week working on porting eSpeak, the bedrock of ~all TTS models, from C++.

It wasn't "obvious" they didn't mean art, and it definitely was not obvious that they want to splice real voice clips at arbitrary points and insert new words without being a detectable fake for a video game. I needed more info to answer. I'm sorry.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#223

Post by lhl »

rahimnathwani wrote: post #181
lhl wrote: post #24

I tested StyleTTS2 last month, my step-by-step notes that might be useful for people doing local setup (not too hard): https://llm-tracker.info/books/howto-guides/page/styletts-2 Also I did a little speed/quality shootoff with the LJSpeech model (vs VITS and XTTS). StyleTTS2 was pretty good and very fast: https://fediverse.randomfoo.net/notice/AaOgprU715gcT5GrZ2

Thanks. Following the instructions now. BTW mamba is no longer recommended (for those like me who aren't already using it), and the #mambaforge anchor in the link didn't work.

I switched from conda to mamba a while ago and never looked back (it's probably saved dozens of hours from waiting for conda's slow as molasses package resolution). I'm looking at the latest docs and it doesn't look like there's any deprecation messages or anything (it does warn against installing mamba inside of conda, but that's been the case for a long time): https://mamba.readthedocs.io/en/latest/installation/mamba-in...

It looks like miniforge is still the recommended install method, but also the anchor has changed in the repo docs, which I've updated, thx. FWIW, I haven't run into any problems using mamba. While I'm not a power user, so there are edge cases I might have missed, but I have over 35 mamba envs on my dev machine atm, so it's definitely been doing the job for me and remains wicked fast (if not particularly disk efficient).

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#224

Post by mx20 »

_lvbh wrote: post #218
deknos wrote: post #204

Is this really opensource and/or free software? like code, data(set/s) and models? I am quite tired to see some "open-source" advertisement, where the half or more is not really free. general psa: please be honest in your announcements :|

MIT licensed. Models, code, and everything is available right there when you click the link. Maybe actually check it out before complaining.

But you are wrong the trained models are separate on Google Drive and have following Text that seems to be an additional License Agreement that also includes using the software and any trained Modell.

License Part 2 Text: "Before using these pre-trained models, you agree to inform the listeners that the speech samples are synthesized by the pre-trained models, unless you have the permission to use the voice you synthesize. That is, you agree to only use voices whose speakers grant the permission to have their voice cloned, either directly or by license before making synthesized voices pubilc, or you have to publicly announce that these voices are synthesized if you do not have the permission to use these voices."

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#225

Post by generalizations »

modeless wrote: post #108
lucubratory wrote: post #104

Earlier quoted context omitted.

How hard on your end does the task of making the chatbot converse naturally look? Specifically I'm thinking about interruptions, if it's talking too long I would like to be able to start talking and interrupt it like in a normal conversation, or if I'm saying something it could quickly interject something. Once you've got the extremely high speed, theoretically faster than real time, you can start doing that stuff ri…

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…

> It could possibly be done zero-shot using prompting of a LLM

That's how I've been thinking of doing it - seemed like you could use a much smaller GPT-J-ish model for that, and measure the relative probability of 'yes' vs 'no' tokens in response to a question like 'is the user done talking'. Seemed like even that would be orders of magnitude better than just waiting for silence.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#226

Post by zestyping »

modeless wrote: post #101

I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…

Wow! For those of us who don't have the necessary GPU hardware, can you post a video?

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#227

Post by washadjeffmad »

nikkwong wrote: post #188
washadjeffmad wrote: post #127

Earlier quoted context omitted.

No, I am. I work with them, and I've been one (am one, rarely). I listed just one possible use, but I also see voice cloning and advanced TTS expanding access for evocative instruction, as an aid to study style and expand range. Don't be afraid on their behalf. The dooming you're talking about applied to every one of the technological changes I already listed, and we employ more performers and artists today than ever…

I don't know, I feel like the work produced through voice acting is more of a commodity than work in the other industries that you're describing. Sure, a voice actor can add a lot of emotion and verbal nuance in a way that is differentiating, but I'm not sure if the difference is enough to matter for most people for the vast majority of cases. (Or I may be too dense to realize it). This is in contradiction to say per…

Where do you see AI being most used in a production pipeline?

Do you think it will replace actors or that it might just reduce the burden on existing talent, like canned audio has done for decades? Will it make ADR easier or cheaper? Will it actually save anyone any money who wants to ever be able to hire a living actor again?

There are a lot of clever sounding, low probability arguments here, and I think a lot of people don't understand the work well enough to identify what are and aren't the elephants.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#228

Post by _lvbh »

mx20 wrote: post #224
_lvbh wrote: post #218

Earlier quoted context omitted.

MIT licensed. Models, code, and everything is available right there when you click the link. Maybe actually check it out before complaining.

But you are wrong the trained models are separate on Google Drive and have following Text that seems to be an additional License Agreement that also includes using the software and any trained Modell. License Part 2 Text: "Before using these pre-trained models, you agree to inform the listeners that the speech samples are synthesized by the pre-trained models, unless you have the permission to use the voice you synth…

That’s for the pre-trained models. Train one up yourself.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#229

Post by regularfry »

modeless wrote: post #157
joshspankit wrote: post #155

Earlier quoted context omitted.

Short-term could it be configured as push to talk?

Certainly, but then it has little advantage over e.g. ChatGPT voice mode. I guess running locally is an advantage but the voice and answer quality is worse. The much better latency and more natural conversation is what I like about it.

What I would really like is a push-to-talk app on my phone, so I can talk to my house from anywhere without worrying about putting a microphone in every room, or monkeying about with wakeword detection. I'm sure it's doable, I'm just not conscious of having seen it done.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#230

Post by a2128 »

KolmogorovComp wrote: post #214
wczekalski wrote: post #55

Earlier quoted context omitted.

One thing I've seen done for style cloning is a high quality fine tuned TTS -> RVC pipeline to "enhance" the output. TTS for intonation + pronunciation, RVC for voice texture. With StyleTTS and this pipeline you should get close to ElevenLabs.

RVC? R… Voice Model?

Retrieval-based Voice Conversion - https://github.com/RVC-Project/Retrieval-based-Voice-Convers...
Post reply on HN