Live data from Hacker News

WhisperSpeech – An open source text-to-speech system built by inverting Whisper

github.com

31–40 of 119 posts

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#31

What’s the text to speech generator that chatGPT uses? It’s the most impressive one I’ve heard so far.

If you think OpenAI's TTS is impressive, you should check out Eleven Labs. They have the highest quality models IMO. Voice quality, emotional awareness / inflection and support for foreign languages are top-notch, it's that last point that OpenAI seems to have the most issues with. If you find a good voice to clone, the latest models can even replicate somewhat unusual accents and speaking styles. For plain old Engli…

The quality difference between Eleven and OpenAI is IMO pretty small, but the price difference is enormous: for 50,000 characters (approx 1hr of audio, by Eleven's estimates), you'd pay Eleven Labs $9 assuming you're in their highest $330/month payment commitment tier; for OpenAI there's no minimum commitment and the same number of characters would cost $0.75.

If you're generating speech once and replaying it many times (e.g. making podcasts), the difference is negligible and you might as well go with Eleven Labs, since it's more customizable and possibly slightly higher quality. If you're doing interactive speech with customers, $9/hr is incredibly expensive (higher than hiring a minimum-wage worker in the U.S.!), and OpenAI's TTS is a very close second best and much more reasonably priced. If you're trying to integrate speech into an AI product, Eleven makes your hourly costs pretty unfeasible since you have to at minimum charge your customers more than it costs to hire a human being to do a task.

Azure's "Neural" line of TTS is the best of the big cloud offerings, but it's pretty mediocre compared to either OpenAI or Eleven Labs IMO. And it's actually more expensive than using OpenAI: it's $0.80 for 50,000 characters (~1hr), unless you're willing to commit to over $1k monthly spend, at which point it's barely cheaper than OpenAI at $0.64 per 50k characters.

OpenAI's TTS is IMO the best option for anything interactive, since it's so much higher quality than Azure's Neural TTS and so much cheaper (with very little quality difference) as compared to Eleven Labs.

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#33
post #28

I've been following jpc [0] on the LAION discord since he started building this last year, and it's a very impressive project. The key here is that the Whisper multilingual ASR model has been trained on a huge amount of data, so its encoder output is a very good representation of the semantic content of speech. This can be used as an open-source, drop-in replacement for the semantic encoder in model architectures lik…

They don't mention the ability to add custom voices to the speech output, I wonder if that's a feature thatbwould be supported

They do mention voice cloning in the README ("We’ve also added an example of voice cloning based on a reference audio file."), do you have something different in mind?

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#34
post #7

Earlier quoted context omitted.

Have you released your flashcard app?

If you're interested, I have a small side project ( https://imaginanki.com ) for generating Anki decks with images + speech (via SDXL/Azure).

Some language learning resources From "Show HN: Open-source tool for creating courses like Duolingo" (2023) https://news.ycombinator.com/item?id=38317345 :

> ENH: Generate Anki decks with {IPA symbols, Greek letters w/ LaTeX for math and science,

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#35
post #29
post #4

I know it's old at this point and doesn't use the fancy new tech, but Mycroft's Mimic 3 is still pretty impressive and is small enough to fit comfortably and generate speech in real time on a raspberry pi [0]. Some of their voices are better than others, but the best of them are definitely equal to the examples of WhisperSpeech given here. [0] https://mycroft.ai/mimic-3/

The "English US" voice sounds more Scottish than American to me :P

Which might be a good thing, nae, laddie?

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#36
post #7
post #6

Interested to see how it performs for Mandarin Chinese speech synthesis, especially with prosody and emotion. The highest quality open source model I've seen so far is EmotiVoice[0], which I've made a CLI wrapper around to generate audio for flashcards.[1] For EmotiVoice, you can apparently also clone your own voice with a GPU, but I have not tested this.[2] [0] https://github.com/netease-youdao/EmotiVoice [1] https:…

Have you released your flashcard app?

Not OP, but I develop Mochi [0] which is a spaced repetition flash card app that has text-to-speech and a bunch of other stuff built in (transcription, dictionaries, etc.) that you might be interested in.

[0] https://mochi.cards

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#37

Earlier quoted context omitted.

on the contrary I'm really disappointed in how long its taking anything to get into production. Whisper and self-hostable LLMs had a cambrian explosion about 1 year ago, I attended a GPT4 hackathon last March and in 48 hours saw people hook up Speech2Text -> LLM -> Text2Speech pipelines for their live demos. I thought we would all have babelfish by June. Months later I later attended some conferences with internation…

I built this last March. It captures audio from a live HLS stream and transcribes and translates into 18 languages on the fly. Used by a customer with about 25K international employees for their internal events. Works surprisingly well.

Fabulous, guess that's the other part of productizing: a paying customer!

Re: WhisperSpeech – An open source text-to-speech system built by inverting Whisper

#39
post #9
post #6

Interested to see how it performs for Mandarin Chinese speech synthesis, especially with prosody and emotion. The highest quality open source model I've seen so far is EmotiVoice[0], which I've made a CLI wrapper around to generate audio for flashcards.[1] For EmotiVoice, you can apparently also clone your own voice with a GPU, but I have not tested this.[2] [0] https://github.com/netease-youdao/EmotiVoice [1] https:…

Did you try XTTS v2 for Mandarin? I'm curious how it compares with EmotiVoice.

It has a big problem with hallucination in Chinese, random extra syllables all over the place.
Post reply on HN