Making Music and Art Through Machine Learning

Re: the latest Performance RNN, discussing their "best sounding" output.

>Note that this isn’t a performance of an existing piece; the model is also choosing the notes to play, “composing” a performance directly. The performances generated by the model lack the overall coherence that one might expect from a piano composition; in musical jargon, it might sound like the model is “noodling”— playing without a long-term structure. However, to our ears, the local characteristics of the performance (i.e. the phrasing within a one or two second time window) are quite expressive.

It's understandable to be encouraged by progress, but to my ears, it's not really expressive other than the first thought that I had when listening to the piece:

"That piano player is having a stroke, somebody should do something to help."

Part of me agrees, but at the same time computers can explore spaces far faster than we can. Perhaps it will end up like Go where an AI exploring music will lead to new inspiration for humans.

I think we're only at the place where the computer can propose ideas for the artist to dispose of mostly, and perhaps find a gem occasionally.

Music is indeed a language, so while it evolves, it still requires a network of meaning to rely upon. So doing something wild like music in 5/6 time will for the most part, bounce off people's heads and not resonate with meaning.

IMO, the end game of this sort of thing would be to, one day, have an app that can generate endless new music just for you, based on your taste.

My take (as someone working in the area) on how/why we are encouraged by this progress:

The model is unconditional i.e. any concept of form at all was learned directly from data, with no real structural hints to what it should learn.

Generally you get much stronger "global structure" the more prior information you provide either in the model itself or in the input/targets (such as chord constraints, etc.). This is one reason that harmonization tasks often sound notably better than pure generation - the "backbone" in harmonization was human provided, whereas in pure generation the model needs to be self-consistent.

The analogous task in language would be character level language modeling, whereas something like harmonization or conditional generation from chords would be more akin to machine translation. Character level language models may be a bit wonky looking when generating, but any structure at all was purely model discovered from a simple rule (generally, maximum likelihood), whereas in conditional models a lot of extra help is given by the conditioning variable.

This model is a pretty big jump in quality from previous efforts in the same vein of unconditional generation, and should combine well with techniques for conditioning in sequence models from all over the research community (translation, speech recognition, question answering, summarization, speech synthesis). On top of that, the model is IMO quite elegant, and much simpler to understand than many other attempts at polyphonic generation including my own.

It is worth comparing this output to some previous efforts (from Magenta and others), some with more complicated internal structure such as [0][1][2][3][4][5][6], and also some models with stronger conditioning or different input representations such as [7][8][9], the extreme case being [10] where a human (Benoît Carré) used a modeling/ML toolkit along with standard arrangement and production approaches to produce a wacky, awesome pop song.

Magenta is putting out lots of neat stuff around interactive usage of these models, with web demos and plugins to standard music tools [11]. Indeed, Doug's quotes in this article [12] sum it up for me, excerpt: “I don’t think that machines themselves just making art for art’s sake is as interesting as you might think,” he explained. “The question to ask is can machines help us make a new kind of art?”

Link to the model in question: Performance RNN, Magenta https://magenta.tensorflow.org/performance-rnn

[0] Work from Doug in his postdoc doing LSTM Blues, http://people.idsia.ch/~juergen/blues/

[1] RNN-RBM from Boulanger-Lewandowski et. al, see TF impl http://danshiebler.com/2016-08-17-musical-tensorflow-part-tw...

[2] Early Magenta models, https://magenta.tensorflow.org/2016/07/15/lookback-rnn-atten... , ex https://www.youtube.com/watch?v=nHxr9u9_4_s

[3] Daniel Johnson's Biaxial RNN http://www.hexahedria.com/2015/08/03/composing-music-with-re...

[4] Some experiments only published in my thesis to date, https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx... , thesis in question https://github.com/kastnerkyle/udem_masters_thesis

[5] Polyphonic Magenta model https://www.youtube.com/watch?v=s0BVFVqEY4A

[6] RL tuning to improve generation https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-n...

[7] Folk-RNN is similar but uses ABC representation (https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recu...), https://soundcloud.com/seaandsailor/sets/char-rnn-composes-i..., also see related post https://maraoz.com/2016/02/02/abc-rnn/

[8] DeepBach , http://www.flow-machines.com/deepbach-polyphonic-music-gener...

[9] Lawson He's LSTM approach, http://www.lawsonhe.com/music.html

[10] Daddy's Car https://www.youtube.com/watch?v=LSHZ_b05W7o

[11] Interactive Magenta Jam https://magenta.tensorflow.org/2016/12/16/nips-demo

[12] https://www.technologyreview.com/s/604010/google-brain-wants...