101–110 of 153 posts
I'm guessing DeepMind has already done this (or is already doing), but conditioning on a video is the obvious next step. It would be incredibly interesting to see how accurate it can get generating the audio for a movie. Though I imagine for really great results they'll need to mix in an adversarial network.
Please please please someone please share an IPython notebook with something working already :)
Earlier quoted context omitted.
I can hear some distortion in the piano notes - which may be an audio compression artefact, or it may be the output of the resynthesis process. If you train NNs at the phrase level and overfit, then you get something that is indeed more or less the same as cross-fading at random between short sections. Piano music is very idiomatic, so you'll capture some typical piano gestures that way. But I'd be surprised if the m…
The samples are uncompressed WAV files, so everything you hear is a direct result of the synthesis process. Some of the distortion is a result of the 16kHz sample rate-- it's not 44.1kHz CD quality.
Earlier quoted context omitted.
Not really. They're training directly on the waveform, so the model can learn intonation. They just need to train on longer samples, and perhaps augment their linguistic representation with some extra discourse analysis. A big problem with generating prosody has always been that our theories of it don't really provide a great prediction of people's behaviours. It's also very expensive to get people to do the prosody…
theres 0 chance of effective intonation and tone without understanding of the material
What would require understanding would be novel arrangements and metaphor to indicate emotional state. On the fly variations to avoid mononticity might also be difficult, as well as sarcasm or combinations/levels (e.g. she spoke matter of factly but with mirth lightly woven through).
The music examples are utterly fascinating. It sounds insanely natural. The only thing I can hear that sounds unnatural, is the way that the reverberation in the room (the "echo") immediately gets lower when the raw piano sound itself gets lower. In a real room, if you produce a loud sound and immediately after a soft sound, the reverberation of the loud sound remains. But since this network only models "the sound ri…
That seems completely tractable by simply adding a bit of the right reverb to the generated sample, more or less "in post".
Earlier quoted context omitted.
theres 0 chance of effective intonation and tone without understanding of the material
And who says it can't understand the material? There have been recurrent networks trained that can translate between languages, or predict the next word in a sentence, at remarkable accuracy. Combined with wavenet this could be quite effective.
Earlier quoted context omitted.
The samples are uncompressed WAV files, so everything you hear is a direct result of the synthesis process. Some of the distortion is a result of the 16kHz sample rate-- it's not 44.1kHz CD quality.
It's quantized to just 256 values though, which could be causing some of the distortion.
Earlier quoted context omitted.
To my Australian English ears, the babbling sounded vaguely Scandinavian.
Indeed. I was surprised by that as well. Sounded like a Dutch speaker with a muffled voice behind a screen.