So when I get the AI from one place, train it with the voices of hundreds of people from dozens of other sources, and then have it read a book from Project Gutenberg to an mp3... who owns the mechanical rights to that recording?
WaveNet: A Generative Model for Raw Audio
11–20 of 153 posts
Re: WaveNet: A Generative Model for Raw Audio
#12Please please please someone please share an IPython notebook with something working already :)
Re: WaveNet: A Generative Model for Raw Audio
#13Please please please someone please share an IPython notebook with something working already :)
I have some iPython notebooks for speech analysis using a Chinese corpus. I used those for a tutorial on machine learning with Python and unfortunately they are still a bit incomplete, but maybe you find them useful nevertheless (no deep learning involved though). What I do in the tutorial is to start from a WAV file and then go through all the steps required for analyzing the data (using a "traditional" approach), i.e. generate the Mel-Cepstrum coefficients of the segmented audio data and then train a model to distinguish individual words. Word segmentation is another topic that I touch a bit, and where we can also use machine learning to improve the results.
Here's a version with very simple speech training data (basically just different syllables with different tones):
https://github.com/adewes/machine-learning-chinese/blob/mast...
More complex speech training data (from a real-world Chinese speech corpus [not included but downloadable]):
https://github.com/adewes/machine-learning-chinese/blob/mast...
There are other parts of the tutorial that deal with Chinese text and character recognition as well, if you're interested:
https://github.com/adewes/machine-learning-chinese
For part 2 I also train a simple neural network with lasagne (a Python library for deep learning), and I plan to add more deep learning content and do a clean write-up of the whole thing as soon as I have some more time.
Re: WaveNet: A Generative Model for Raw Audio
#14This can be used to implement seamless voice performance transfer from one speaker to another:
1. Train a WaveNet with the source speaker.
2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the target WaveNet.
3. Record raw audio from the source speaker.
Fun fact: any algorithmic process that "renders" something given a set of inputs can be "run in reverse" to recover those inputs given the rendered output. In this case, we now have raw audio from the source speaker that—in principle— could have been rendered by the source speaker's WaveNet, and we want to recover the inputs that would have rendered it, had we done so.
To do that, usually you convert all numbers in the forward renderer into Dual numbers and use automatic differentiation to recover the inputs (in this case, phonemes and what not).
4. Recover the inputs. (This is computationally expensive, but not difficult in practice, especially if WaveNet's generation algorithm is implemented in C++ and you've got a nice black-box optimizer to apply to the inputs, of which there are many freely available options.)
5. Take the recovered WaveNet inputs, feed them into the target speaker's WaveNet, and record the resulting audio.
Result: The resulting raw audio will have the same overall performance and speech as the source speaker, but rendered completely naturally in the target speaker's voice.
Re: WaveNet: A Generative Model for Raw Audio
#15This is incredible. I'd be worried if I were a professional audiobook reader :)
I wouldn't. The results they offer are excellent, but the missing points they need to achieve human level are related to producing the correct intonation, which requires accurate understanding of the material. That is still at least ten years in the future, I expect.
Re: WaveNet: A Generative Model for Raw Audio
#16Please please please someone please share an IPython notebook with something working already :)
I have some iPython notebooks for speech analysis using a Chinese corpus. I used those for a tutorial on machine learning with Python and unfortunately they are still a bit incomplete, but maybe you find them useful nevertheless (no deep learning involved though). What I do in the tutorial is to start from a WAV file and then go through all the steps required for analyzing the data (using a "traditional" approach), i…
Thanks! will take a look.
Re: WaveNet: A Generative Model for Raw Audio
#17I hope this shows up as a TTS option for VoiceDream (http://www.voicedream.com/) soon! With the best voices they have to offer (currently, the ones from Ivona), I can suffer through a book if the subject is really interesting, but the way the samples sounded here, the WaveNet TTS could be quite pleasant to listen to.
Re: WaveNet: A Generative Model for Raw Audio
#18This can be used to implement seamless voice performance transfer from one speaker to another: 1. Train a WaveNet with the source speaker. 2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the target WaveNet . 3. Record raw audio from the source speaker. Fun fact: any algorithmic process that "re…
Basically the same idea as style transfer with image algorithms. Looking forward to Abraham Lincoln reading audiobooks to me.
Re: WaveNet: A Generative Model for Raw Audio
#19This is incredible. I'd be worried if I were a professional audiobook reader :)
I wouldn't. The results they offer are excellent, but the missing points they need to achieve human level are related to producing the correct intonation, which requires accurate understanding of the material. That is still at least ten years in the future, I expect.
I don't see why many aspects of intonation couldn't be taught the same way ...
Re: WaveNet: A Generative Model for Raw Audio
#20This can be used to implement seamless voice performance transfer from one speaker to another: 1. Train a WaveNet with the source speaker. 2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the target WaveNet . 3. Record raw audio from the source speaker. Fun fact: any algorithmic process that "re…
Maybe I'm reading this paper incorrectly, but it seems that in this system "voice" is part of the model parameters not inputs. What they did was train the same model with multiple reader voices while using one of the inputs to keep track of which voice the model was currently trained on. So the model can switch between different voices, but only between those which it was trained on.
"The conditioning was applied by feeding the speaker ID to the model in the form of a one-hot vector. The dataset consisted of 44 hours of data from 109 different speakers."
Am I missing something?