Yes, please integrate it with Mistral and Whisper. This has got to get into the LLM frontends.
StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
121–130 of 245 posts
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#122I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…
How do you get Whisper to be fast?
Isn't it quite non-realtime?
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#123The weights aren’t MIT-licensed, so this is not usable in commercial applications, right?
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#124I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…
How do you get Whisper to be fast? Isn't it quite non-realtime?
The community upgrades to whisper are far faster than real-time, especially if you have a powerful gpu
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#125I tested StyleTTS2 last month, my step-by-step notes that might be useful for people doing local setup (not too hard): https://llm-tracker.info/books/howto-guides/page/styletts-2 Also I did a little speed/quality shootoff with the LJSpeech model (vs VITS and XTTS). StyleTTS2 was pretty good and very fast: https://fediverse.randomfoo.net/notice/AaOgprU715gcT5GrZ2
> inferences at up to 15-95X (!) RT on my 4090 That's incredible! Are infill and outpainting equivalents possible? Super-RT TTS at this level of quality opens up a diverse array of uses esp for indie/experimental gamedev that I'm excited for.
>Are infill and outpainting equivalents possible?
Do you mean outpainting as in you still what words to do, or the model just extends the audio unconditionally the way some image models just expand past an image borders without a specific prompt (in audio like https://twitter.com/jonathanfly/status/1650001584485552130)
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#126Earlier quoted context omitted.
How hard on your end does the task of making the chatbot converse naturally look? Specifically I'm thinking about interruptions, if it's talking too long I would like to be able to start talking and interrupt it like in a normal conversation, or if I'm saying something it could quickly interject something. Once you've got the extremely high speed, theoretically faster than real time, you can start doing that stuff ri…
Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…
you'd have to do something along the lines of what voice comm does to combat the output feedback problem. i think it involves an fft to analyse the two signals and cancel out the feedback, im not 100% sure on the details.
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#127Earlier quoted context omitted.
Hardly. Imagine licensing your voice to Amazon so that any customer could stream any book narrated in your likeness without you having to commit the time to record. You could still work as a custom voice artist, all with a "no clone" clause if you chose. You could profit from your performance and craft in a fraction of the time, focusing as your own agent on the management of your assets. Or, you could just keep and…
What you're not considering here is that a large majority of this industry is made up of no-name voice actors who have a pleasant (but perfectly substitutible) voice which is now something that AI can do perfectly and at a fraction of the price. Sure, celebrities and other well-known figures will have more to gain here as they can license out their voice; but the majority of voice actors won't be able to capitalize o…
No, I am. I work with them, and I've been one (am one, rarely).
I listed just one possible use, but I also see voice cloning and advanced TTS expanding access for evocative instruction, as an aid to study style and expand range.
Don't be afraid on their behalf. The dooming you're talking about applied to every one of the technological changes I already listed, and we employ more performers and artists today than ever in history.
When animation went digital, we graduated more storyboard artists and digital animators. When music notation software and sampling could replace musicians and orchestras, we graduated more musicians and composers trained on those tools. Now it's the performing arts, and no one in industry is going to shrink their pool of available talent (or risk ire) by daring conflate authenticity and performance with virtual impersonation. Performance capture and vfx also didn't kill or consolidate the movie industry - it allowed it to expand.
Art evolves, and so does its business. People who love art want to see people who do art succeed. I'm optimistic.
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#128I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…
How do you get Whisper to be fast? Isn't it quite non-realtime?
use whisper-distil, it's like 5-8x faster
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#129Is it possible to optimize somehow the model to run a Raspberry with 4 GB of RAM?
[deleted]
Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
#130Is it possible to optimize somehow the model to run a Raspberry with 4 GB of RAM?
I was able to get it work with libjemalloc.