Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WhisperSpeech – An open source text-to-speech system built by inverting Whisper

github.com

Post reply on HN
Print view View original

91–100 of 119 posts

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#91

Post by jpcl »

dale_glass wrote: post #70
jpcl wrote: post #53

Hi, WhisperSpeech dev here. Thanks for all the nice comments, I was working really hard on this model for quite a few months now but there are still a lot of ways we can make it better. Thanks to generosity of Collabora this is a real open-source project (not just a one-time marketing ploy), so if you want to help improve it or integrate it into something you are building, I'd love to help. You can also buy our undiv…

> Thanks to generosity of Collabora this is a real open-source project (not just a one-time marketing ploy), so if you want to help improve it or integrate it into something you are building, I'd love to help. We're probably interested! We're Overte, an open source VR/Desktop social platform. The system targets VR and voice chat primarily, but we want to be more accessible to people who can't use voice chat for any r…

Yeah, we'd love to help you when you decide to give it a try so feel free to reach out. We also have quite a few people working on VR at Collabora.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#92

Post by jpcl »

yorwba wrote: post #76
jpcl wrote: post #68

Earlier quoted context omitted.

For Polish I have around 700hr. I suspect that we will need less hours if we add more languages since they do overlap to some extent. Fixed transcripts would be nice although we need to align them with the audio really precisely (we cut the audio into 30 second chunks and we pretty much need to have the exact text in every chunk). It seems this can be solved with forced alignment algorithms but I have not dived into…

I have forced alignments, too. E.g. for the True Story of Ah Q https://github.com/Yorwba/LiteratureForEyesAndEars/tree/mast... .align.json is my homegrown alignment format, .srt are standard subtitles, .txt is the text, but note that in some places I have [[original text||what it is pronounced as]] annotations to make the forced alignment work better. (E.g. the "." in LibriVox.org, pronounced as 點 "diǎn" in Mandarin.…

Thanks, I'll check it out. I don't know any Chinese so I'll probably reach out to you for some help :)

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#93

Post by jpcl »

e12e wrote: post #82
jpcl wrote: post #51

Earlier quoted context omitted.

Both Polish and English samples are actually synthesized with a voice trained on the WolneLektury audiobooks. They are the highest quality open source (CC BY-SA) audiobooks I could find. By using the Whisper-derived phonetic representation (so called semantic tokens) we successfully trained a model with just a high-quality speech dataset of one language and the voice quality transferred to English.

Link to these in English? I found some hits that may be correct for Polish - but I'm guessing they're hosted somewhere canonical?

https://wolnelektury.pl/katalog/audiobooki/ is the Polish audiobook collection.

The English audiobooks are public domain recordings from LibriVox (via the LibriLight dataset).

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#94

Post by yorwba »

jpcl wrote: post #92
yorwba wrote: post #76

Earlier quoted context omitted.

I have forced alignments, too. E.g. for the True Story of Ah Q https://github.com/Yorwba/LiteratureForEyesAndEars/tree/mast... .align.json is my homegrown alignment format, .srt are standard subtitles, .txt is the text, but note that in some places I have [[original text||what it is pronounced as]] annotations to make the forced alignment work better. (E.g. the "." in LibriVox.org, pronounced as 點 "diǎn" in Mandarin.…

Thanks, I'll check it out. I don't know any Chinese so I'll probably reach out to you for some help :)

Sure, feel free to email me at the address included in every commit in the repo.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#95

Post by BeetleB »

ametrau wrote: post #89
BeetleB wrote: post #79

Earlier quoted context omitted.

For anyone reading, in case you want a whole order of magnitude cheaper, just go with Google Cloud TTS. For many voices, you get 1 million characters free per month, and even beyond that it's ridiculously cheap. Some voices do sound artificial, but many sound quite human - the only tells are the relatively consistent tone and section ends (no appropriate pauses). I don't read long articles any more. I have a script t…

None of their available voices are as good as ms

ms?

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#96

Post by BeetleB »

stavros wrote: post #88
BeetleB wrote: post #79

Earlier quoted context omitted.

For anyone reading, in case you want a whole order of magnitude cheaper, just go with Google Cloud TTS. For many voices, you get 1 million characters free per month, and even beyond that it's ridiculously cheap. Some voices do sound artificial, but many sound quite human - the only tells are the relatively consistent tone and section ends (no appropriate pauses). I don't read long articles any more. I have a script t…

That's a good suggestion, thank you. Would it be possible to post some code? I've found GCP's APIs/documentation to be a bit abstruse.

Actually, all the code related to this was copied from the docs, almost verbatim.

(But yes, their docs suck).

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#97

Post by doctorpangloss »

leereeves wrote: post #71
jpcl wrote: post #54

Earlier quoted context omitted.

Yeah, Whisper is not clear-cut but since it is not a generative model I think their data usage is a lot more likely to be considered fair-use. And the part of that which we use for WhisperSpeech is just the phonetic representation so our model is not able to recreate any of the Whisper training data in any way.

The readme says "We are working only with properly licensed speech recordings and all the code is Open Source so the model will be always safe to use for commercial applications." Is that less certain than the quote implies?

Laypeople value the aesthetics of statements like these. It's very Discord energy.

Everyone using learned weights from other models, especially ones released by OpenAI, Stability and Google, such as text and audio encoders, is tainted by training materials that were not expressly licensed for the purpose of AI model training or unlimitedly licensed for any use.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#98

Post by jpcl »

freedomben wrote: post #83
jpcl wrote: post #50

Earlier quoted context omitted.

Yeah, the Mimic is a lot less resource intensive. We are working to improve WhisperSpeech in this regard but it's probably always going to require more compute (but in return you'll get higher quality). That said if you have a modern NVidia GPU you should be able to run a voice-bot in real-time with WhisperSpeech.

Will something like whisper.cpp be possible for whisper speech?

We looked at this at one point and it seems whisper.cpp/llama.cpp have all the bits needed to make it work. I'd love to help if someone wanted to give it a shot.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#99

Post by jpcl »

doctorpangloss wrote: post #97
leereeves wrote: post #71

Earlier quoted context omitted.

The readme says "We are working only with properly licensed speech recordings and all the code is Open Source so the model will be always safe to use for commercial applications." Is that less certain than the quote implies?

Laypeople value the aesthetics of statements like these. It's very Discord energy. Everyone using learned weights from other models, especially ones released by OpenAI, Stability and Google, such as text and audio encoders, is tainted by training materials that were not expressly licensed for the purpose of AI model training or unlimitedly licensed for any use.

That's true but you make it sound like it's totally obvious where the line of fair use should be drawn for AI training.

Until courts or lawmakers make it clearer I personally believe non-generative models (Whisper, ResNet, DINOv2) should be legally trainable on publicly released data. Generative models (image or video generation, TTS, LLMs?) should be held to a much higher scrutiny since their outputs can potentially compete the creators who put a lot of creativity into their art. That not true for an ImageNet-trained classification model or Whisper ASR.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#100

Post by rhdunn »

Is there any work/progress on a NN/trained model based on Internet Phonetic Alphabet (IPA) transcriptions? I.e. to be able to create an IPA transcription and convert it back to sound.

That approach would be useful for things like shifting a voice to a different accent and to support voices speaking multiple languages.

This can be done to a limited extend for models such as MBROLA voices by mapping the phonemes of one language to the phonemes of the MBROLA voice. MBROLA is more complex in that it supports diphones, and many diphone pairs don't exist, so you need to map 3 phonemes together to get the best matching phonetic transcription.

The IPA approach may also make it better to train the phonetic synthesis, given that the IPA vowels are in a formant continuum (similar to colour wheels and cubes). Then, the model could better learn the variations in voice quality and tambre.

Post reply on HN