Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WhisperSpeech – An open source text-to-speech system built by inverting Whisper

github.com

Post reply on HN
Print view View original

111–119 of 119 posts

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#111

Post by reissbaker »

BeetleB wrote: post #79
reissbaker wrote: post #31

Earlier quoted context omitted.

The quality difference between Eleven and OpenAI is IMO pretty small, but the price difference is enormous: for 50,000 characters (approx 1hr of audio, by Eleven's estimates), you'd pay Eleven Labs $9 assuming you're in their highest $330/month payment commitment tier; for OpenAI there's no minimum commitment and the same number of characters would cost $0.75. If you're generating speech once and replaying it many ti…

For anyone reading, in case you want a whole order of magnitude cheaper, just go with Google Cloud TTS. For many voices, you get 1 million characters free per month, and even beyond that it's ridiculously cheap. Some voices do sound artificial, but many sound quite human - the only tells are the relatively consistent tone and section ends (no appropriate pauses). I don't read long articles any more. I have a script t…

Azure has a half-million character free tier for their top-quality "Neural" voices, which I find somewhat better than Google Cloud's top tier of voices ("Studio" voices). For personal use you can probably just use Azure for free too!

If you're running a business you'll probably burn through the free tiers of either of them, and Google is wayyy more expensive — roughly $8/hr for Studio voices using the 50k characters per hour estimate. The "Neural2" voices are competitively priced with OpenAI and Azure, but are pretty low quality compared to even Azure (and much worse than OpenAI).

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#112

Post by jpcl »

doctorpangloss wrote: post #106
jpcl wrote: post #99

Earlier quoted context omitted.

That's true but you make it sound like it's totally obvious where the line of fair use should be drawn for AI training. Until courts or lawmakers make it clearer I personally believe non-generative models (Whisper, ResNet, DINOv2) should be legally trainable on publicly released data. Generative models (image or video generation, TTS, LLMs?) should be held to a much higher scrutiny since their outputs can potentially…

I believe your use should be protected. This is not meant to be a takedown, better you hear it from me though, because you'll never hear it from Discord. > "We are working only with properly licensed" ...versus: > "fair use" You're smart enough to know these are very different things - saying you believe you are protected by fair use, and claiming that the data is "properly licensed." In legal there is a colossal dif…

Yeah, thanks. I'd love to try to clarify this (I'll put this into our documentation as well ASAP) for anyone that may be reading this in the future:

Our model is not a derivative of Whisper but we do use the transcripts (and encoder outputs) in our data preprocessing. I am convinced this data cannot be regarded as a derivative of the (proprietary) Whisper training set. Whisper itself is open-source (MIT) and has no special TOS (unlike, say, ChatGPT).

WhisperSpeech itself is trained on mostly Public Domain and some CC-BY-SA recordings. We are working on adding more data and we will provide clearer documentation on all the licenses.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#113

Post by jpcl »

e12e wrote: post #107
jpcl wrote: post #93

Earlier quoted context omitted.

https://wolnelektury.pl/katalog/audiobooki/ is the Polish audiobook collection. The English audiobooks are public domain recordings from LibriVox (via the LibriLight dataset).

Thank you. Is the Polish collection also a volunteer effort? Link to librivox for others: https://librivox.org/

Not really, the Polish effort is run by a non-profit and hired professional voice actors.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#114

Post by reissbaker »

BeetleB wrote: post #95
ametrau wrote: post #89

Earlier quoted context omitted.

None of their available voices are as good as ms

ms?

MS = Microsoft (presumably the Azure Neural TTS, which I agree is better than Google's TTS, although worse than OpenAI).

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#115

Post by kholk »

jpcl wrote: post #67
WhackyIdeas wrote: post #64

Earlier quoted context omitted.

And totally private, as in no internet needed?

Yes, you download the weights once from Huggingface and you can do whatever you want with it. :) We have no cloud APIs or usage tracking of any kind.

Since it's 12x faster than real time on a 4090, I wonder how fast would it be on a small form factor device (a SBC); I get it as this is using CUDA, so I really wonder how would that perform on my nV Xavier NX (and the more common Nano's out there)...!
Post reply on HN