Can it run local only?
WhisperSpeech – An open source text-to-speech system built by inverting Whisper
21–30 of 119 posts
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#22Aside: is it just me, or is anyone else just as dumbfounded with how quickly literally every aspect of AI and LLMs and Models and blah blah blah is going? Am I weird in just having my head spin - even though I've also been at leading edge tech before, but this is just me yelling at these new algos on my lawn?
on the contrary I'm really disappointed in how long its taking anything to get into production. Whisper and self-hostable LLMs had a cambrian explosion about 1 year ago, I attended a GPT4 hackathon last March and in 48 hours saw people hook up Speech2Text -> LLM -> Text2Speech pipelines for their live demos. I thought we would all have babelfish by June. Months later I later attended some conferences with internation…
I built this last March. It captures audio from a live HLS stream and transcribes and translates into 18 languages on the fly. Used by a customer with about 25K international employees for their internal events. Works surprisingly well.
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#23I know it's old at this point and doesn't use the fancy new tech, but Mycroft's Mimic 3 is still pretty impressive and is small enough to fit comfortably and generate speech in real time on a raspberry pi [0]. Some of their voices are better than others, but the best of them are definitely equal to the examples of WhisperSpeech given here. [0] https://mycroft.ai/mimic-3/
If you're not already aware, the primary developer of Mimic 3 (and its non-Mimic predecessor Larynx) continued TTS-related development with Larynx and the renamed project Piper: https://github.com/rhasspy/piper
Last year Piper development was supported by Nabu Casa for their "Year of Voice" project for Home Assistant and it sounds like Mike Hansen is going to continue on it with their support this year.
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#24Aside: is it just me, or is anyone else just as dumbfounded with how quickly literally every aspect of AI and LLMs and Models and blah blah blah is going? Am I weird in just having my head spin - even though I've also been at leading edge tech before, but this is just me yelling at these new algos on my lawn?
on the contrary I'm really disappointed in how long its taking anything to get into production. Whisper and self-hostable LLMs had a cambrian explosion about 1 year ago, I attended a GPT4 hackathon last March and in 48 hours saw people hook up Speech2Text -> LLM -> Text2Speech pipelines for their live demos. I thought we would all have babelfish by June. Months later I later attended some conferences with internation…
> I'm really disappointed in how long its taking anything to get into production.
> It was humbling and also hype-busting to realize that it takes time to productize
Yep, looks like you found out why it’s taking so long to get this new tech into production. The gap between nothing and a proof of concept is, in some ways, much smaller than the gap between proof of concept and commercial product.
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#25What’s the text to speech generator that chatGPT uses? It’s the most impressive one I’ve heard so far.
If you think OpenAI's TTS is impressive, you should check out Eleven Labs. They have the highest quality models IMO. Voice quality, emotional awareness / inflection and support for foreign languages are top-notch, it's that last point that OpenAI seems to have the most issues with. If you find a good voice to clone, the latest models can even replicate somewhat unusual accents and speaking styles.
For plain old English TTS with a stock voice, there isn't that much of a difference (although Eleven Labs still wins IMO), but if you need either voice cloning or foreign language support, nothing else comes even close.
With that said, Eleven is extremely pricy, something like Azure TTS (which is the best among the cheap options) may be a better fit for less demanding applications.
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#26What’s the text to speech generator that chatGPT uses? It’s the most impressive one I’ve heard so far.
If you think OpenAI's TTS is impressive, you should check out Eleven Labs. They have the highest quality models IMO. Voice quality, emotional awareness / inflection and support for foreign languages are top-notch, it's that last point that OpenAI seems to have the most issues with. If you find a good voice to clone, the latest models can even replicate somewhat unusual accents and speaking styles. For plain old Engli…
Maybe I’m not a good judge but OpenAI’s voices sound very natural to me and seem better than Eleven labs.
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#27Earlier quoted context omitted.
on the contrary I'm really disappointed in how long its taking anything to get into production. Whisper and self-hostable LLMs had a cambrian explosion about 1 year ago, I attended a GPT4 hackathon last March and in 48 hours saw people hook up Speech2Text -> LLM -> Text2Speech pipelines for their live demos. I thought we would all have babelfish by June. Months later I later attended some conferences with internation…
I'd be interested if you ever dig anything up for this. I hacked together a kind of crude tool to snapshot audio and translate / caption it on the fly: https://captioner.richardson.co.nz/ I would very much like to improve on this but the live translation / captioning still has some more work to go in this space. Source was here: https://github.com/Rodeoclash/captioner
I was going to suggest considering looking into vosk but... clearly that suggestion isn't very useful to you. :)
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#28I've been following jpc [0] on the LAION discord since he started building this last year, and it's a very impressive project. The key here is that the Whisper multilingual ASR model has been trained on a huge amount of data, so its encoder output is a very good representation of the semantic content of speech. This can be used as an open-source, drop-in replacement for the semantic encoder in model architectures lik…
They don't mention the ability to add custom voices to the speech output, I wonder if that's a feature thatbwould be supported
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#29I know it's old at this point and doesn't use the fancy new tech, but Mycroft's Mimic 3 is still pretty impressive and is small enough to fit comfortably and generate speech in real time on a raspberry pi [0]. Some of their voices are better than others, but the best of them are definitely equal to the examples of WhisperSpeech given here. [0] https://mycroft.ai/mimic-3/
The "English US" voice sounds more Scottish than American to me :P
Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper
#30Interested to see how it performs for Mandarin Chinese speech synthesis, especially with prosody and emotion. The highest quality open source model I've seen so far is EmotiVoice[0], which I've made a CLI wrapper around to generate audio for flashcards.[1] For EmotiVoice, you can apparently also clone your own voice with a GPU, but I have not tested this.[2] [0] https://github.com/netease-youdao/EmotiVoice [1] https:…
Just listened to the demo voices for EmotiVoice and WhisperSpeech. I think WhisperSpeech edges out EmotiVoice. EmotiVoice sounds like it was trained on English spoken by non-native speakers.