WhisperSpeech – An open source text-to-speech system built by inverting Whisper

Post by freedomben » Thu, Jan 18, 2024, 3:54 PM UTC

yorwba wrote: post #65

jpcl wrote: post #43

Earlier quoted context omitted.

Hi, WhisperSpeech dev here, we only support Polish and English at the moment but we just finished doing some inference optimizations and are looking to add more languages. What we seem to need is high-quality speech recordings in any language (audiobooks are great) and some recordings for each target language which can be low-quality but need varied prosody/emotions (otherwise everything we generate will sound like a…

Last I checked, LibriVox had about 11 hours of Mandarin audiobooks and Common Voice has 234 validated hours of "Chinese (China)" (probably corresponding to Mandarin as spoken on the mainland paired with text in Simplified characters, but who knows) and 77 validated hours of "Chinese (Taiwan)" (probably Taiwanese Mandarin paired with Traditional characters). Not sure whether that's enough data for you. (If you need pa…

Post by freedomben » Thu, Jan 18, 2024, 3:59 PM UTC

jpcl wrote: post #50

lolinder wrote: post #4

I know it's old at this point and doesn't use the fancy new tech, but Mycroft's Mimic 3 is still pretty impressive and is small enough to fit comfortably and generate speech in real time on a raspberry pi [0]. Some of their voices are better than others, but the best of them are definitely equal to the examples of WhisperSpeech given here. [0] https://mycroft.ai/mimic-3/

Yeah, the Mimic is a lot less resource intensive. We are working to improve WhisperSpeech in this regard but it's probably always going to require more compute (but in return you'll get higher quality). That said if you have a modern NVidia GPU you should be able to run a voice-bot in real-time with WhisperSpeech.

Post by pksebben » Thu, Jan 18, 2024, 4:30 PM UTC

globalnode wrote: post #40

this is the best tts ive heard, the voice modulates as you'd expect a human to.

Post by pksebben » Thu, Jan 18, 2024, 4:34 PM UTC

jazzyjackson wrote: post #13

samstave wrote: post #10

Aside: is it just me, or is anyone else just as dumbfounded with how quickly literally every aspect of AI and LLMs and Models and blah blah blah is going? Am I weird in just having my head spin - even though I've also been at leading edge tech before, but this is just me yelling at these new algos on my lawn?

on the contrary I'm really disappointed in how long its taking anything to get into production. Whisper and self-hostable LLMs had a cambrian explosion about 1 year ago, I attended a GPT4 hackathon last March and in 48 hours saw people hook up Speech2Text -> LLM -> Text2Speech pipelines for their live demos. I thought we would all have babelfish by June. Months later I later attended some conferences with internation…

Post by ametrau » Thu, Jan 18, 2024, 5:00 PM UTC

BeetleB wrote: post #79

reissbaker wrote: post #31

Earlier quoted context omitted.

The quality difference between Eleven and OpenAI is IMO pretty small, but the price difference is enormous: for 50,000 characters (approx 1hr of audio, by Eleven's estimates), you'd pay Eleven Labs $9 assuming you're in their highest $330/month payment commitment tier; for OpenAI there's no minimum commitment and the same number of characters would cost $0.75. If you're generating speech once and replaying it many ti…

For anyone reading, in case you want a whole order of magnitude cheaper, just go with Google Cloud TTS. For many voices, you get 1 million characters free per month, and even beyond that it's ridiculously cheap. Some voices do sound artificial, but many sound quite human - the only tells are the relatively consistent tone and section ends (no appropriate pauses). I don't read long articles any more. I have a script t…