Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WhisperSpeech – An open source text-to-speech system built by inverting Whisper

github.com

Post reply on HN
Print view View original

101–110 of 119 posts

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#101

Post by jpcl »

rhdunn wrote: post #100

Is there any work/progress on a NN/trained model based on Internet Phonetic Alphabet (IPA) transcriptions? I.e. to be able to create an IPA transcription and convert it back to sound. That approach would be useful for things like shifting a voice to a different accent and to support voices speaking multiple languages. This can be done to a limited extend for models such as MBROLA voices by mapping the phonemes of one…

That's an interesting thought. The semantic tokens we get from Whisper serve a similar purpose – you can convert existing speech to different voices, I did not try with accents yet.

There is still a lot to explore in this space – we certainly don't have all the answers yet!

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#102

Post by jpcl »

stavros wrote: post #86
londons_explore wrote: post #80

The first demo on that page was trained from a 32kbps crappy sound quality clip of winston churchill...? garbage in, garbage out?

Must have been, it sounds very much like the quality of the "we shall fight on the beaches" speech. A bit unfortunate choice for a demo, sadly.

Good point, thanks. And I was thinking it will show that the model can really synthesize very varied samples... ;)

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#103

Post by stavros »

jpcl wrote: post #102
stavros wrote: post #86

Earlier quoted context omitted.

Must have been, it sounds very much like the quality of the "we shall fight on the beaches" speech. A bit unfortunate choice for a demo, sadly.

Good point, thanks. And I was thinking it will show that the model can really synthesize very varied samples... ;)

It does, but maybe put it last!

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#104

Post by jpcl »

stavros wrote: post #103
jpcl wrote: post #102

Earlier quoted context omitted.

Good point, thanks. And I was thinking it will show that the model can really synthesize very varied samples... ;)

It does, but maybe put it last!

Did exactly that, thanks for spotting that. :)

https://github.com/collabora/WhisperSpeech/commit/398b889060...

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#105

Post by RockRobotRock »

Havoc wrote: post #58
RockRobotRock wrote: post #32

holy shit what. how?

Same way the algos designed to spot cats in pictures can generate pictures of cats in reverse. Sorta

Open source text to speech has been really lagging, so I'm really happy there's another tool out there.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#106

Post by doctorpangloss »

jpcl wrote: post #99
doctorpangloss wrote: post #97

Earlier quoted context omitted.

Laypeople value the aesthetics of statements like these. It's very Discord energy. Everyone using learned weights from other models, especially ones released by OpenAI, Stability and Google, such as text and audio encoders, is tainted by training materials that were not expressly licensed for the purpose of AI model training or unlimitedly licensed for any use.

That's true but you make it sound like it's totally obvious where the line of fair use should be drawn for AI training. Until courts or lawmakers make it clearer I personally believe non-generative models (Whisper, ResNet, DINOv2) should be legally trainable on publicly released data. Generative models (image or video generation, TTS, LLMs?) should be held to a much higher scrutiny since their outputs can potentially…

I believe your use should be protected. This is not meant to be a takedown, better you hear it from me though, because you'll never hear it from Discord.

> "We are working only with properly licensed"

...versus:

> "fair use"

You're smart enough to know these are very different things - saying you believe you are protected by fair use, and claiming that the data is "properly licensed." In legal there is a colossal difference, you went from say you were authorized to use something to you believe you are unauthorized but still permitted due to fair use.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#107

Post by e12e »

jpcl wrote: post #93
e12e wrote: post #82

Earlier quoted context omitted.

Link to these in English? I found some hits that may be correct for Polish - but I'm guessing they're hosted somewhere canonical?

https://wolnelektury.pl/katalog/audiobooki/ is the Polish audiobook collection. The English audiobooks are public domain recordings from LibriVox (via the LibriLight dataset).

Thank you. Is the Polish collection also a volunteer effort?

Link to librivox for others: https://librivox.org/

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#108

Post by thorum »

jpcl wrote: post #68
yorwba wrote: post #65

Earlier quoted context omitted.

Last I checked, LibriVox had about 11 hours of Mandarin audiobooks and Common Voice has 234 validated hours of "Chinese (China)" (probably corresponding to Mandarin as spoken on the mainland paired with text in Simplified characters, but who knows) and 77 validated hours of "Chinese (Taiwan)" (probably Taiwanese Mandarin paired with Traditional characters). Not sure whether that's enough data for you. (If you need pa…

For Polish I have around 700hr. I suspect that we will need less hours if we add more languages since they do overlap to some extent. Fixed transcripts would be nice although we need to align them with the audio really precisely (we cut the audio into 30 second chunks and we pretty much need to have the exact text in every chunk). It seems this can be solved with forced alignment algorithms but I have not dived into…

You might check out this list from espnet. They list the different corpuses they use to train their models sorted by language and task (ASR, TTS etc):

https://github.com/espnet/espnet/blob/master/egs2/README.md

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#109

Post by mastayoda »

jpcl wrote: post #53

Hi, WhisperSpeech dev here. Thanks for all the nice comments, I was working really hard on this model for quite a few months now but there are still a lot of ways we can make it better. Thanks to generosity of Collabora this is a real open-source project (not just a one-time marketing ploy), so if you want to help improve it or integrate it into something you are building, I'd love to help. You can also buy our undiv…

Great job! Thanks for sharing this!

I'm developing a dynamic tutorial desktop app and I plan to use this model as text to speech synthesizer. Any chance it can be ported to ONNX format?

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#110

Post by jazzyjackson »

Havoc wrote: post #57
huijzer wrote: post #18

Earlier quoted context omitted.

> on the contrary I'm really disappointed in how long its taking anything to get into production. I agree. I was thinking about making a Jarvis like bot which should be pretty easy at this point. The main problem was that my iPhone doesn’t easily allow for pressing a button upon which it starts listening. You always need to unlock first at which the whole screen gets unlocked too. Maybe these kind of GUI-focussed int…

The tap on back might work without unlock and i think that can be set to a custom shortcut

smart, I just played around with it, I can use the shortcuts app to create an action (like turn off the lamp or 'identify music via shazam') and then go to accessibility->touch settings and use double tap to trigger the action.

It works without unlocking but the phone has to be awake, so I can hit the power button once to wake and then double tap the back to trigger a shortcut without unlocking, confirmed to work without needing to identify me via face ID or anything.

There is also an accessibility shortcut for triple-clicking the side button but it only allows for toggling accessibility features.

Post reply on HN