Amazing, how to make this happen?
GANs + "A ton of python magic" [1] :)
[1] https://twitter.com/xsteenbrugge/status/1188798045158293505
Visualizing Music with GANs
11–20 of 34 posts
Re: Visualizing Music with GANs
#12Earlier quoted context omitted.
This may be GANs as the author stated but the end result looks surprisingly similar to a bunch of pixel shaders making transitions between source and target images with said transitions driven either by pure algorithms and/or derived from blurred images themselves. I've implemented music visualizer ages ago using similar concepts (pure algos though, no real images). It happened when nVidia released the first affordab…
Cool! How'd you make your video dance to a beat? Is the code available anywhere? @jonathanfly has been posting some interesting machine learning effects: https://twitter.com/jonathanfly/status/1185843103271444480 I guess one way to make these effects dance to music would be to make a Mel spectrogram of the audio, then somehow use the shapes in the spectrogram to apply deltas to the rendered frames.
The music animation code worked something like this:
1) Each of my pixel shaders was driven by let's say 32 parameters (do not remember the exact value)
2) The code would generate first set of said parameters and the second set (values were random) and start transitioning ( lerp ) between 2 with the length of transition of about 60 seconds.
3) Upon completion of the transition the first set would be replaced by second set and the second set would be replaced by freshly generated third set and an infinum.
4) Seps 2 and 3 allowed for non stop fluid motion.
5) Lerp value the degree of transition between sets for each parameter would be modulated by sound (one FFT band for each also passed through synth like attack / decay.
6) Finally there was beat detection part which upon detecting a beat would invert lerp direction
There were more steps and various tricks to make it more interesting and non repetitive but I am not writing article here ;)
The end result was quite artistic. The visualizer was part of much bigger enterprise grade media playback / management / delivery /scheduling platform I've developed for hospitality industry
Re: Visualizing Music with GANs
#13I had trouble connecting the music to the transitions and morphs. I perceived it just as music overlaid on fun-to-see GAN-generated images, though clearly that’s not the aim. The music is beautiful and the imagery is intriguing, however.
This is something I've been working on https://www.youtube.com/watch?v=52qWiLoOeIQ
you should clearly see the connection between the morphs and the music
Re: Visualizing Music with GANs
#14I had trouble connecting the music to the transitions and morphs. I perceived it just as music overlaid on fun-to-see GAN-generated images, though clearly that’s not the aim. The music is beautiful and the imagery is intriguing, however.
I don't really get what the connection to the music is supposed to be either. Does the music influence the GAN somehow?
Re: Visualizing Music with GANs
#15I had trouble connecting the music to the transitions and morphs. I perceived it just as music overlaid on fun-to-see GAN-generated images, though clearly that’s not the aim. The music is beautiful and the imagery is intriguing, however.
I don't really get what the connection to the music is supposed to be either. Does the music influence the GAN somehow?
You may notice that the visual changes occur at the same frequency as the music's tempo. But you're right, there is no influence on the content itself.
Re: Visualizing Music with GANs
#16I had trouble connecting the music to the transitions and morphs. I perceived it just as music overlaid on fun-to-see GAN-generated images, though clearly that’s not the aim. The music is beautiful and the imagery is intriguing, however.
yeah i when i read the title i was hoping the GAN visualizations would reveal the underlying structure of the music somehow, or the images would have a more compelling link to the music. EDIT: just realized it might be much more interesting to train GANs with movie soundtracks, thus forming a link between music and image
Re: Visualizing Music with GANs
#17I had trouble connecting the music to the transitions and morphs. I perceived it just as music overlaid on fun-to-see GAN-generated images, though clearly that’s not the aim. The music is beautiful and the imagery is intriguing, however.
This is something I've been working on https://www.youtube.com/watch?v=52qWiLoOeIQ you should clearly see the connection between the morphs and the music
I'm blown away.
So the glasses are always present when the bass/808s are hitting, so is there something that maps the sound to the images?
What is it about the algorithms that make the images 'dance' so quickly between the 3.5 beat and the 1? Is it because there are static risers that move so quickly through the wave spectrum?
Wait... is light skin mapped to when highs dominate and dark skin to lows?
Re: Visualizing Music with GANs
#18Earlier quoted context omitted.
This is something I've been working on https://www.youtube.com/watch?v=52qWiLoOeIQ you should clearly see the connection between the morphs and the music
I'm blown away. So the glasses are always present when the bass/808s are hitting, so is there something that maps the sound to the images? What is it about the algorithms that make the images 'dance' so quickly between the 3.5 beat and the 1? Is it because there are static risers that move so quickly through the wave spectrum? Wait... is light skin mapped to when highs dominate and dark skin to lows?
I'm glad you like it! Actually compared to the linked post I don't do any manual latent-space representation selection. It's just a bit of "smart" signal processing. I've written a framework that makes it really simple to do these visualizations (not open-source yet). Here's one more example: https://www.youtube.com/watch?v=X4r4njUjE2M
Re: Visualizing Music with GANs
#19Earlier quoted context omitted.
I don't really get what the connection to the music is supposed to be either. Does the music influence the GAN somehow?
You may notice that the visual changes occur at the same frequency as the music's tempo. But you're right, there is no influence on the content itself.
That could be accomplished by tying the speed of the animation playback to the tempo.
Re: Visualizing Music with GANs
#20Is there shareable / repeatable code to go behind it? Its cool but without seeing how it was produced, it could be just as well a fancy output of a video editing software