Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Neural text to speech with dozens of celebrity voices

vocodes.com

Post reply on HN
Print view View original

71–80 of 174 posts

Re: Show HN: Neural text to speech with dozens of celebrity voices

#71

Post by echelon »

blueblisters wrote: post #48
echelon wrote: post #22

Earlier quoted context omitted.

I trained a base model on the Linda Johnson speech (LJS) data set for several days. I then transfer learned for each of these speakers. Some speakers have as little as 40 minutes of data, others have up to five hours. The resulting quality isn't strictly a function of the amount of training data, though more typically helps. It's also important to have high fidelity text transcriptions free of errors. The transfer le…

Have you had any success with using speaker embeddings to generate voices with fewer samples of speech? I did some cursory experiments but I couldn't get too far beyond getting pitch similar to the target speaker. My reasoning for this approach: IMO, if the model learns a "universal human voice", it shouldn't need too much additional information to get a target voice.

I did! I tried creating a multi-speaker embedding model for practical concerns: saving on memory costs. I'm going to have to add additional layers, because it didn't fit individual speakers very well. I wish I'd saved audio results to share. I might be able to publish my findings if I look around for the model files.

I think you're right in that if we can get such a model to work, training new embeddings won't require much data.

Re: Show HN: Neural text to speech with dozens of celebrity voices

#73

Post by echelon »

calebkaiser wrote: post #56
echelon wrote: post #13

Earlier quoted context omitted.

I can make a blog post later, but at a high level: A rust TTS server hosts two models: a mel inference model and a mel inversion model. The ones I'm using are glow-tts and melgan. They fit together back to back in a pipeline. I chose these models not for their fidelity, but for their performance. They're 10x faster at inference than Tacotron 2. If you want something that sounds amazing, you're better off with a dense…

This is incredibly cool. Do you mind sharing how big the models are, and what kind of instances you're deploying them on? I ask because I help maintain an open source ML infra project ( https://github.com/cortexlabs/cortex ) and we've recently done a lot of work around autoscaling multi-model endpoints. Always curious to see how others are approaching this.

glow-tts:

    total 4.2G
    -rw-r--r-- 1 bt bt 110M glow-tts_alan-rickman_ljstx_2020.07.22_expr-1_chkpt-4765.torchjit
    -rw-r--r-- 1 bt bt 110M glow-tts_anderson_cooper_ljstx_2020.07.21_expr-1_chkpt-6622.torchjit
    -rw-r--r-- 1 bt bt 110M glow-tts_arnold_schwarzenegger_ljstx_2020.07.16_expr-2_chkpt-9045.torchjit
    -rw-r--r-- 1 bt bt 110M glow-tts_barack_obama_ljstx_2020.06.28_expr-1_chkpt-1729.torchjit
    -rw-r--r-- 1 bt bt 110M glow-tts_ben-stein_ljstx_2020.07.21_expr-1_chkpt-7516.torchjit
    -rw-r--r-- 1 bt bt 110M glow-tts_betty_white_ljstx_2020.06.28_expr-1_chkpt-1666.torchjit
    ...
melgan:

    -rw-r--r-- 1 bt bt 17M melgan_manyvoice5.0_2020-07-23_12d5838_10760.torchjit
(All the voices use the same melgan, or derivations of it.)

I'll edit my post later with my deployment and cluster architecture. In short, it's sharded and proxied from a thin microservice at the top of the stack. I'll probably introduce a job queue soon.

Re: Show HN: Neural text to speech with dozens of celebrity voices

#74

Post by nmstoker »

echelon wrote: post #66
spdustin wrote: post #63

Earlier quoted context omitted.

> "…Instead of using graphemes, I'm using ARPABET phonemes…" Is this why some examples I tried seemed to skip some of the words?

Exactly. If you type "I am a dangerous asdhfjahdsff velociraptor, rawr." There aren't entries for - asdhfjahdsff - rawr I added around 500 new words, but I missed a lot of stuff. The ultimate fix is to have grapheme -> phoneme prediction so that all unseen words can be mapped to potential phonemes (polyphones).

Are you logging the words people submit? That'd be a good source for the most common OOV tokens to add.

Re: Show HN: Neural text to speech with dozens of celebrity voices

#75

Post by echelon »

mywittyname wrote: post #43
echelon wrote: post #11

Earlier quoted context omitted.

Absolutely. These are called "style tokens" and they're an active area of TTS research. The problem is that currently your training data has to be annotated with these tokens, and that adds a lot to the difficulty of creating data sets. I imagine that over time this will get much easier to do.

Are there good emotion detectors for speech-to-text? Much like they have for facial recognition?

I'm not aware of any, and I haven't had much time to look as I'm not to the point of doing style tokens yet. I'm certain this would be useful for annotating data and for all sorts of other applications. Sentiment analysis, etc.

Re: Show HN: Neural text to speech with dozens of celebrity voices

#78

Post by nmstoker »

echelon wrote: post #61
maxerickson wrote: post #18

It doesn't like spelling mistakes. Ask it to say " The Aristoocrats!" with Gilbert Goddfried.

It's currently sourcing phonemes from a lookup table called CMUdict, which is constructed by Carnegie Mellon [1]. That database has 140,000 entries, but even so, you'd be surprised how many common words are omitted. And of course it is missing terms for things like "pokemon" and "fortnite", which I had to add myself. I don't have generic grapheme -> phoneme/polyphone prediction, but that's something I look to add soo…

Espeak-ng has pretty decent English word to phoneme translation. You run it in the mode where it just outputs the IPA. The vocabulary can be extended too (as the coverage is good but far from perfect)

Re: Show HN: Neural text to speech with dozens of celebrity voices

#80

Post by tmoney1818 »

echelon wrote: post #28
ethbro wrote: post #25

Earlier quoted context omitted.

Out of curiosity, are there legal concerns? E.g. training off Schwarzenegger and offering an Arnold transform

I'm not a lawyer, but I think we're entering into a legal gray area. There are the existing frameworks of copyright, parody, free speech, slander, libel, etc. that are all somewhat tangential to this. I believe (I'm not certain) that celebrity voice impersonation is legal as long as it is not used to sell or endorse a product. Most models are trained on the original speaker's voice, but maybe only a little bit. Model…

>"Most models are trained on the original speaker's voice, but maybe only a little bit."

Really cool that you got this to work. I used to work on TTS (a few years ago, now), and we trained on celebrity voices, but used full audiobooks. https://github.com/Kyubyong/tacotron

Here are some of our Nick Offerman samples: https://soundcloud.com/kyubyong-park/sets/tacotron_nick_215k .

Post reply on HN