Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WhisperSpeech – An open source text-to-speech system built by inverting Whisper

github.com

Post reply on HN
Print view View original

61–70 of 119 posts

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#61

Post by IshKebab »

follower wrote: post #23
lolinder wrote: post #4

I know it's old at this point and doesn't use the fancy new tech, but Mycroft's Mimic 3 is still pretty impressive and is small enough to fit comfortably and generate speech in real time on a raspberry pi [0]. Some of their voices are better than others, but the best of them are definitely equal to the examples of WhisperSpeech given here. [0] https://mycroft.ai/mimic-3/

If you're not already aware, the primary developer of Mimic 3 (and its non-Mimic predecessor Larynx) continued TTS-related development with Larynx and the renamed project Piper: https://github.com/rhasspy/piper Last year Piper development was supported by Nabu Casa for their "Year of Voice" project for Home Assistant and it sounds like Mike Hansen is going to continue on it with their support this year.

That is much better quality than Mimic (which didn't sound very close to WhisperSpeech to me).

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#62

Post by jpcl »

satvikpendem wrote: post #60
jpcl wrote: post #51

Earlier quoted context omitted.

Both Polish and English samples are actually synthesized with a voice trained on the WolneLektury audiobooks. They are the highest quality open source (CC BY-SA) audiobooks I could find. By using the Whisper-derived phonetic representation (so called semantic tokens) we successfully trained a model with just a high-quality speech dataset of one language and the voice quality transferred to English.

How much training compute does it require to train from scratch? I'm wondering because I have a lot of audiobooks, they're not necessarily CC licensed though but for my private usage and training I think it'd be fine.

Training the T2S model from scratch takes around 8h on 96 A100 GPUs. Training the `tiny` S2A model is around 3x faster (training HQ `small` variant is comparable to T2S).

I think you would get good results with fine-tuning but unfortunately we don't have a user-friendly notebook or script to do that right now. The biggest model is 800MB (FP32) so you won't even need a very big GPU to be able to fine-tune.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#63

Post by jpcl »

deskamess wrote: post #55
jpcl wrote: post #49

Earlier quoted context omitted.

Yup, we are using Whisper to transcribe automatically so we can train the model on just speech recordings, without human transcripts. This works for any language that is well supported by the OpenAI Whisper model.

Where can we find the latest OpenAI language model rankings?

There is a plot of language performance on their repo: https://github.com/openai/whisper

I am not aware of a multi-lingual leaderboard for speech recognition models.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#64

Post by WhackyIdeas »

jpcl wrote: post #46
WhackyIdeas wrote: post #21

Can it run local only?

Yes, on a consumer 4090 card it's 12x faster than real-time. We'll benchmark some older cards as well for comparison. I think it should work pretty good with the Apple's MLX framework as well if anyone would be willing to convert it. :)

And totally private, as in no internet needed?

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#65

Post by yorwba »

jpcl wrote: post #43
siraben wrote: post #6

Interested to see how it performs for Mandarin Chinese speech synthesis, especially with prosody and emotion. The highest quality open source model I've seen so far is EmotiVoice[0], which I've made a CLI wrapper around to generate audio for flashcards.[1] For EmotiVoice, you can apparently also clone your own voice with a GPU, but I have not tested this.[2] [0] https://github.com/netease-youdao/EmotiVoice [1] https:…

Hi, WhisperSpeech dev here, we only support Polish and English at the moment but we just finished doing some inference optimizations and are looking to add more languages. What we seem to need is high-quality speech recordings in any language (audiobooks are great) and some recordings for each target language which can be low-quality but need varied prosody/emotions (otherwise everything we generate will sound like a…

Last I checked, LibriVox had about 11 hours of Mandarin audiobooks and Common Voice has 234 validated hours of "Chinese (China)" (probably corresponding to Mandarin as spoken on the mainland paired with text in Simplified characters, but who knows) and 77 validated hours of "Chinese (Taiwan)" (probably Taiwanese Mandarin paired with Traditional characters).

Not sure whether that's enough data for you. (If you need paired text for the LibriVox audiobooks, I can provide you with versions where I "fixed" the original text to match the audiobook content e.g. when someone skipped a line.)

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#66

Post by jpcl »

dale_glass wrote: post #59
jpcl wrote: post #45

Earlier quoted context omitted.

We support voice cloning so you can mimic the sound of any real voice (or try to create random ones). The prosody/emotions are more difficult to control right now but we are looking into this. To check how this works in practice you can check the Google Collab link, at the end we are cloning the voice from a Churchill's speech over radio.

Sounds excellent! What are the requirements to run this regarding hardware? How much VRAM? Does it work on AMD or Intel Arc?

Both models are using around 3GB right now (converted into FP16 for speed). But I checked that the (slower) FP32 version uses 2.3GB so we are probably doing something suboptimal here.

We support CUDA right now although it should not be too hard to port it to whisper/llama.cpp or Apple's MLX. It's a pretty straightforward transformer architecture.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#67

Post by jpcl »

WhackyIdeas wrote: post #64
jpcl wrote: post #46

Earlier quoted context omitted.

Yes, on a consumer 4090 card it's 12x faster than real-time. We'll benchmark some older cards as well for comparison. I think it should work pretty good with the Apple's MLX framework as well if anyone would be willing to convert it. :)

And totally private, as in no internet needed?

Yes, you download the weights once from Huggingface and you can do whatever you want with it. :) We have no cloud APIs or usage tracking of any kind.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#68

Post by jpcl »

yorwba wrote: post #65
jpcl wrote: post #43

Earlier quoted context omitted.

Hi, WhisperSpeech dev here, we only support Polish and English at the moment but we just finished doing some inference optimizations and are looking to add more languages. What we seem to need is high-quality speech recordings in any language (audiobooks are great) and some recordings for each target language which can be low-quality but need varied prosody/emotions (otherwise everything we generate will sound like a…

Last I checked, LibriVox had about 11 hours of Mandarin audiobooks and Common Voice has 234 validated hours of "Chinese (China)" (probably corresponding to Mandarin as spoken on the mainland paired with text in Simplified characters, but who knows) and 77 validated hours of "Chinese (Taiwan)" (probably Taiwanese Mandarin paired with Traditional characters). Not sure whether that's enough data for you. (If you need pa…

For Polish I have around 700hr. I suspect that we will need less hours if we add more languages since they do overlap to some extent.

Fixed transcripts would be nice although we need to align them with the audio really precisely (we cut the audio into 30 second chunks and we pretty much need to have the exact text in every chunk). It seems this can be solved with forced alignment algorithms but I have not dived into that yet.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#69

Post by lioeters »

colechristensen wrote: post #11
samstave wrote: post #10

Aside: is it just me, or is anyone else just as dumbfounded with how quickly literally every aspect of AI and LLMs and Models and blah blah blah is going? Am I weird in just having my head spin - even though I've also been at leading edge tech before, but this is just me yelling at these new algos on my lawn?

This is the structure of revolutions, particularly of this kind. Exponential growth looks like this. In particular with the generation / recognition abilities of ML models, they have this feature of being a curiosity but not quite useful... so if a speech recognition program goes from 50% accuracy to 75% accuracy it's a huge accomplishment but the program is still approximately as useless when it's done. Going from 9…

> structure of revolutions

For anyone who hasn't heard of it, this phrase is a reference to the theory of paradigm shifts in scientific progress, introduced in the book "The Structure of Scientific Revolutions" by Thomas Kuhn.

https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#70

Post by dale_glass »

jpcl wrote: post #53

Hi, WhisperSpeech dev here. Thanks for all the nice comments, I was working really hard on this model for quite a few months now but there are still a lot of ways we can make it better. Thanks to generosity of Collabora this is a real open-source project (not just a one-time marketing ploy), so if you want to help improve it or integrate it into something you are building, I'd love to help. You can also buy our undiv…

> Thanks to generosity of Collabora this is a real open-source project (not just a one-time marketing ploy), so if you want to help improve it or integrate it into something you are building, I'd love to help.

We're probably interested! We're Overte, an open source VR/Desktop social platform.

The system targets VR and voice chat primarily, but we want to be more accessible to people who can't use voice chat for any reason. We do have an integrated chat, but it's not an ideal experience in VR. So good TTS to make it integrate better would be great for us. And the possibility of doing this without some sort of commercial API that requires keeping a secret API key is huge.

So yeah, we're very much interested in giving this one a try. It will probably take some time as we're gearing up for FOSDEM now, though.

Post reply on HN