Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Neural text to speech with dozens of celebrity voices

vocodes.com

Post reply on HN
Print view View original

11–20 of 174 posts

Re: Show HN: Neural text to speech with dozens of celebrity voices

#11

Post by echelon »

mattigames wrote: post #9

I was wondering, wouldn't it be possible to classify the voice of every celebrity based on moods so one could make the voices less monotonic? So one could then add text metadata for the text-to-speech conversion, e.g. "[Angry] I have a dream, [Calm] but it has a patent so you can't copy it! (laughter) [Calm-fade-to-angry] In reality insomnia took it from me!"

Absolutely. These are called "style tokens" and they're an active area of TTS research.

The problem is that currently your training data has to be annotated with these tokens, and that adds a lot to the difficulty of creating data sets.

I imagine that over time this will get much easier to do.

Re: Show HN: Neural text to speech with dozens of celebrity voices

#12

Post by imjared »

echelon wrote: post #10
programmarchy wrote: post #5

Looks awesome, but haven't been able to get a result back yet. I think you may be getting hugged to death :)

That's odd. I'm testing it right now and it's working. Which voices are you trying, and which device and browser are you using?

I've tried Gilbert Gottfried and NDT. I do get a console error about CORS: > Access to fetch at 'https://mumble.stream/speak_spectrogram' from origin 'https://vo.codes' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Using Chrome stable

Re: Show HN: Neural text to speech with dozens of celebrity voices

#13

Post by echelon »

Firerouge wrote: post #3

I skimmed your about, where you mention it as a hobby demo of your deep work. Do you have a GitHub or technical documentation about how you build this sort of thing to work at scale?

I can make a blog post later, but at a high level:

A rust TTS server hosts two models: a mel inference model and a mel inversion model. The ones I'm using are glow-tts and melgan. They fit together back to back in a pipeline.

I chose these models not for their fidelity, but for their performance. They're 10x faster at inference than Tacotron 2. If you want something that sounds amazing, you're better off with a denser set of networks, like Tacotron 2 + WaveGlow. You should use these for achieving superior offline results for multimedia purposes.

Instead of using graphemes, I'm using ARPABET phonemes, and I get these from a lookup table called "CMUdict" from Carnegie Mellon. In the future I'll supplement this with a model that predicts phonemes for missing entries.

Each TTS server only hosts one or two voices due to memory constraints. These models are huge. This fleet is scaled horizontally. A proxy server sits in front and decodes the request and directs it to the appropriate backend based on a ConfigMap that associates a service with the underlying model. Kubernetes is used to wire all of this up.

Re: Show HN: Neural text to speech with dozens of celebrity voices

#14

Post by echelon »

imjared wrote: post #12
echelon wrote: post #10

Earlier quoted context omitted.

That's odd. I'm testing it right now and it's working. Which voices are you trying, and which device and browser are you using?

I've tried Gilbert Gottfried and NDT. I do get a console error about CORS: > Access to fetch at ' https://mumble.stream/speak_spectrogram' from origin ' https://vo.codes' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled. Using Chrome stable

Oh man, I thought I had this CORS stuff sorted.

Thanks for the help and info!

I'm using version 84.0.4147.89 (Official Build) (64-bit) and getting back responses.

I got the following response headers:

  access-control-allow-origin: https://vo.codes
  content-length: 151689
  content-type: application/json
  date: Mon, 27 Jul 2020 15:55:37 GMT
  vary: Origin
  x-backend-hostname: tts-group-1-965d444f5-7kvkm
I'll try to dump the cache and reproduce.

edit: I must have an old browser. It works everywhere I'm testing it. CORS is hard. :(

Re: Show HN: Neural text to speech with dozens of celebrity voices

#17

Post by boarnoah »

On a more positive side to this technology.

I've been wondering about the possibility of using this sort of tech (or the API offerings from Azure or GCP) to provide voice overs in video games.

By that I mean for smaller budget Indie development, it would be certainly interesting to either be able to generate voice audio from transcripts in order to add voices to background NPCs and so on (or even the possibility of doing it at run time to produce much more dynamic worlds).

I guess the biggest blocker is the difficulty in conveying emotion with what is currently available as well as the difficulty in getting pronunciation correct (especially with nouns).

Re: Show HN: Neural text to speech with dozens of celebrity voices

#20

Post by lowdose »

sunsetMurk wrote: post #15

Very cool, and easy to use! Can you give some more info on how you generated the models? I'm also interested in the tech stack you're using to implement this webapp... Would love some details! ..What's next?

> What's next?

Text To Video webapp that renders text to video + voice synchronised of famous people.

Who wouldn't like to laugh 5X more when social scrolling?

The first platform that enables creators with the ability to produce deep fakes of celebs from text that they can broadcast as HQ video content to their audience will kill both Youtube & Instagram.

Ranking based on likes so the best jokes of the day are trending on top of the feed.

Recommendation engine with a multibandid ML algo from the start so you can leverage all that incoming data.

Post reply on HN