WhisperSpeech – An open source text-to-speech system built by inverting Whisper

Post by leereeves » Thu, Jan 18, 2024, 2:57 PM UTC

jpcl wrote: post #54

nmfisher wrote: post #20

I've been following jpc [0] on the LAION discord since he started building this last year, and it's a very impressive project. The key here is that the Whisper multilingual ASR model has been trained on a huge amount of data, so its encoder output is a very good representation of the semantic content of speech. This can be used as an open-source, drop-in replacement for the semantic encoder in model architectures lik…

Yeah, Whisper is not clear-cut but since it is not a generative model I think their data usage is a lot more likely to be considered fair-use. And the part of that which we use for WhisperSpeech is just the phonetic representation so our model is not able to recreate any of the Whisper training data in any way.

Post by genpfault » Thu, Jan 18, 2024, 3:11 PM UTC

dale_glass wrote: post #41

How tunable is the voice? I'm interested in applying TTS to a chat system, and one important feature for that is that there should be as many as possible distinct voices, so that each person would have their own. Would this, or something else be able to do that?

Post by lolinder » Thu, Jan 18, 2024, 3:28 PM UTC

follower wrote: post #23

lolinder wrote: post #4

I know it's old at this point and doesn't use the fancy new tech, but Mycroft's Mimic 3 is still pretty impressive and is small enough to fit comfortably and generate speech in real time on a raspberry pi [0]. Some of their voices are better than others, but the best of them are definitely equal to the examples of WhisperSpeech given here. [0] https://mycroft.ai/mimic-3/

If you're not already aware, the primary developer of Mimic 3 (and its non-Mimic predecessor Larynx) continued TTS-related development with Larynx and the renamed project Piper: https://github.com/rhasspy/piper Last year Piper development was supported by Nabu Casa for their "Year of Voice" project for Home Assistant and it sounds like Mike Hansen is going to continue on it with their support this year.

Post by samstave » Thu, Jan 18, 2024, 3:42 PM UTC

jazzyjackson wrote: post #13

samstave wrote: post #10

Aside: is it just me, or is anyone else just as dumbfounded with how quickly literally every aspect of AI and LLMs and Models and blah blah blah is going? Am I weird in just having my head spin - even though I've also been at leading edge tech before, but this is just me yelling at these new algos on my lawn?

on the contrary I'm really disappointed in how long its taking anything to get into production. Whisper and self-hostable LLMs had a cambrian explosion about 1 year ago, I attended a GPT4 hackathon last March and in 48 hours saw people hook up Speech2Text -> LLM -> Text2Speech pipelines for their live demos. I thought we would all have babelfish by June. Months later I later attended some conferences with internation…

Post by BeetleB » Thu, Jan 18, 2024, 3:50 PM UTC

reissbaker wrote: post #31

miki123211 wrote: post #25

Earlier quoted context omitted.

If you think OpenAI's TTS is impressive, you should check out Eleven Labs. They have the highest quality models IMO. Voice quality, emotional awareness / inflection and support for foreign languages are top-notch, it's that last point that OpenAI seems to have the most issues with. If you find a good voice to clone, the latest models can even replicate somewhat unusual accents and speaking styles. For plain old Engli…

The quality difference between Eleven and OpenAI is IMO pretty small, but the price difference is enormous: for 50,000 characters (approx 1hr of audio, by Eleven's estimates), you'd pay Eleven Labs $9 assuming you're in their highest $330/month payment commitment tier; for OpenAI there's no minimum commitment and the same number of characters would cost $0.75. If you're generating speech once and replaying it many ti…