Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WaveNet: A Generative Model for Raw Audio

deepmind.com

Post reply on HN
Print view View original

21–30 of 153 posts

Re: WaveNet: A Generative Model for Raw Audio

#21

Post by swsieber »

grandalf wrote: post #19
espadrine wrote: post #15

Earlier quoted context omitted.

I wouldn't. The results they offer are excellent, but the missing points they need to achieve human level are related to producing the correct intonation, which requires accurate understanding of the material. That is still at least ten years in the future, I expect.

I don't see why many aspects of intonation couldn't be taught the same way ...

I think the point is that different parts of the story need different intonation patterns (reading a scary part vs a boring part, etc.).

So in theory, it could be achieved by having multiple training sets (for the different intonation styles), along with analysis of the text to direct which part of the text needs what intonation. You might even be able to blend intonations.

Re: WaveNet: A Generative Model for Raw Audio

#22

Post by erichocean »

romaniv wrote: post #20
erichocean wrote: post #14

This can be used to implement seamless voice performance transfer from one speaker to another: 1. Train a WaveNet with the source speaker. 2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the target WaveNet . 3. Record raw audio from the source speaker. Fun fact: any algorithmic process that "re…

Maybe I'm reading this paper incorrectly, but it seems that in this system "voice" is part of the model parameters not inputs. What they did was train the same model with multiple reader voices while using one of the inputs to keep track of which voice the model was currently trained on. So the model can switch between different voices, but only between those which it was trained on. "The conditioning was applied by…

These are the "inputs" I'm talking about recovering (from the link):

"In order to use WaveNet to turn text into speech, we have to tell it what the text is. We do this by transforming the text into a sequence of linguistic and phonetic features (which contain information about the current phoneme, syllable, word, etc.) and by feeding it into WaveNet."

The raw audio from Step 3 was (in principle) generated by that input on a properly trained WaveNet. We need to recover that so we can transfer it to the target WaveNet.

How a specific WaveNet instance is configured (as you point out, it's part of the model parameters) is an implementation detail that is irrelevant for the steps I proposed.

Re: WaveNet: A Generative Model for Raw Audio

#23

Post by dhammack »

erichocean wrote: post #14

This can be used to implement seamless voice performance transfer from one speaker to another: 1. Train a WaveNet with the source speaker. 2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the target WaveNet . 3. Record raw audio from the source speaker. Fun fact: any algorithmic process that "re…

It seems like you're using WaveNet to do speech-to-text when we have better tools for that. To transfer text from Trump to Clinton, first run speech-to-text on Trump speech and then give that to a WaveNet trained on Clinton to generate speech that sounds like her but says the same thing as Trump.

Re: WaveNet: A Generative Model for Raw Audio

#24

Post by swsieber »

erichocean wrote: post #14

This can be used to implement seamless voice performance transfer from one speaker to another: 1. Train a WaveNet with the source speaker. 2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the target WaveNet . 3. Record raw audio from the source speaker. Fun fact: any algorithmic process that "re…

Oh, pair this with facial mapping[1] and you pretty much have an "impersonate any famous person" system.

[1] http://www.graphics.stanford.edu/~niessner/thies2016face.htm...

Re: WaveNet: A Generative Model for Raw Audio

#25

Post by erichocean »

dhammack wrote: post #23
erichocean wrote: post #14

This can be used to implement seamless voice performance transfer from one speaker to another: 1. Train a WaveNet with the source speaker. 2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the target WaveNet . 3. Record raw audio from the source speaker. Fun fact: any algorithmic process that "re…

It seems like you're using WaveNet to do speech-to-text when we have better tools for that. To transfer text from Trump to Clinton, first run speech-to-text on Trump speech and then give that to a WaveNet trained on Clinton to generate speech that sounds like her but says the same thing as Trump.

> It seems like you're using WaveNet to do speech-to-text

I'm proposing reducing a vocal performance into the corresponding WaveNet input. At no point in that process is the actual "text" recovered, and doing so would defeat the whole purpose, since I don't care about the text, I care about the performance of speaking the text (whatever it was).

In your example, I can't force Trump to say something in particular. But I can force myself, so I could record myself saying something I wanted Clinton to say [Step 3] (and in a particular way, too!), and if I had a trained WaveNet for myself and Clinton, I could make it seem like Clinton actually said it.

Re: WaveNet: A Generative Model for Raw Audio

#26

Post by bbctol »

Wow! I'd been playing around with machine learning and audio, and this blows even my hilariously far-future fantasies of speech generation out of the water. I guess when you're DeepMind, you have both the brainpower and resources to tackle sound right at the waveform level, and rely on how increasingly-magical your NNs seem to rebuild everything else you need. Really amazing stuff.

Re: WaveNet: A Generative Model for Raw Audio

#27

Post by erichocean »

swsieber wrote: post #24
erichocean wrote: post #14

This can be used to implement seamless voice performance transfer from one speaker to another: 1. Train a WaveNet with the source speaker. 2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the target WaveNet . 3. Record raw audio from the source speaker. Fun fact: any algorithmic process that "re…

Oh, pair this with facial mapping[1] and you pretty much have an "impersonate any famous person" system. [1] http://www.graphics.stanford.edu/~niessner/thies2016face.htm...

Yup, I work in virtual filmmaking and there are tons of way to use this stuff.

I give us 10-15 years before it's not possible to trust anything you see or hear that's recorded.

Re: WaveNet: A Generative Model for Raw Audio

#28

Post by nhaliday »

swsieber wrote: post #21
grandalf wrote: post #19

Earlier quoted context omitted.

I don't see why many aspects of intonation couldn't be taught the same way ...

I think the point is that different parts of the story need different intonation patterns (reading a scary part vs a boring part, etc.). So in theory, it could be achieved by having multiple training sets (for the different intonation styles), along with analysis of the text to direct which part of the text needs what intonation. You might even be able to blend intonations.

Or just pay MTurk workers to annotate texts with intonation cues.

I kinda doubt that would be profitable relative to just hiring readers, but in general you don't need to replace workers completely to cannibalize some of their wages/jobs.

Re: WaveNet: A Generative Model for Raw Audio

#30

Post by drewler »

swsieber wrote: post #21
grandalf wrote: post #19

Earlier quoted context omitted.

I don't see why many aspects of intonation couldn't be taught the same way ...

I think the point is that different parts of the story need different intonation patterns (reading a scary part vs a boring part, etc.). So in theory, it could be achieved by having multiple training sets (for the different intonation styles), along with analysis of the text to direct which part of the text needs what intonation. You might even be able to blend intonations.

Or the author could just add that information to the text. This way there's no need to "understand" it.
Post reply on HN