Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Visualizing Music with GANs

twitter.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Visualizing Music with GANs

#21

Post by isoprophlex »

McIceT wrote: post #18
jressey wrote: post #17

Earlier quoted context omitted.

I'm blown away. So the glasses are always present when the bass/808s are hitting, so is there something that maps the sound to the images? What is it about the algorithms that make the images 'dance' so quickly between the 3.5 beat and the 1? Is it because there are static risers that move so quickly through the wave spectrum? Wait... is light skin mapped to when highs dominate and dark skin to lows?

I'm glad you like it! Actually compared to the linked post I don't do any manual latent-space representation selection. It's just a bit of "smart" signal processing. I've written a framework that makes it really simple to do these visualizations (not open-source yet). Here's one more example: https://www.youtube.com/watch?v=X4r4njUjE2M

Jesus holy fuck that's SUPER IMPRESSIVE

I'd be delighted to learn how you translate musical structure to a point in latent space!

Re: Visualizing Music with GANs

#23

Post by pierrec »

FpUser wrote: post #8
withparadox2 wrote: post #2

Amazing, how to make this happen?

This may be GANs as the author stated but the end result looks surprisingly similar to a bunch of pixel shaders making transitions between source and target images with said transitions driven either by pure algorithms and/or derived from blurred images themselves. I've implemented music visualizer ages ago using similar concepts (pure algos though, no real images). It happened when nVidia released the first affordab…

Regarding the music synchronization, OK, this is ancient stuff.

However, in terms of graphics, this strikes me as different from anything that was possible before the recent advances in GANs. During the era you're talking about, the art of shader-based music visualizers was being pushed by projects like Milkdrop 2, and nowadays a lot of similar research still happens on Shadertoy, and the demoscene, of course, hasn't stopped blowing people's minds.

But this is on another level entirely. It's as is the content and seemingly human concepts themselves are being smoothly animated.

Re: Visualizing Music with GANs

#24

Post by JonathanFly »

McIceT wrote: post #18
jressey wrote: post #17

Earlier quoted context omitted.

I'm blown away. So the glasses are always present when the bass/808s are hitting, so is there something that maps the sound to the images? What is it about the algorithms that make the images 'dance' so quickly between the 3.5 beat and the 1? Is it because there are static risers that move so quickly through the wave spectrum? Wait... is light skin mapped to when highs dominate and dark skin to lows?

I'm glad you like it! Actually compared to the linked post I don't do any manual latent-space representation selection. It's just a bit of "smart" signal processing. I've written a framework that makes it really simple to do these visualizations (not open-source yet). Here's one more example: https://www.youtube.com/watch?v=X4r4njUjE2M

Looking forward to when you release this, would love to try this with other models!

Re: Visualizing Music with GANs

#25

Post by jcims »

McIceT wrote: post #18
jressey wrote: post #17

Earlier quoted context omitted.

I'm blown away. So the glasses are always present when the bass/808s are hitting, so is there something that maps the sound to the images? What is it about the algorithms that make the images 'dance' so quickly between the 3.5 beat and the 1? Is it because there are static risers that move so quickly through the wave spectrum? Wait... is light skin mapped to when highs dominate and dark skin to lows?

I'm glad you like it! Actually compared to the linked post I don't do any manual latent-space representation selection. It's just a bit of "smart" signal processing. I've written a framework that makes it really simple to do these visualizations (not open-source yet). Here's one more example: https://www.youtube.com/watch?v=X4r4njUjE2M

It could just be my brain, but it seems like there is a loose correlation between the mouths in the video and the lyrics.

In Phantom Part II they mostly have their mouths closed. In La La Land it varies but the mouths are mostly open. If you focus on the mouth you'll get little mental radar blips where the mouth could be tracking what is being said.

"Pretty girl and you let go" - https://youtu.be/52qWiLoOeIQ?t=18

"If you wanna waste time baby" - https://youtu.be/52qWiLoOeIQ?t=44

"Yeah i met her at a one oak" - https://youtu.be/52qWiLoOeIQ?t=84 (esp the one oak part)

Anyone that actually watches these will probably just think I'm low on sleep, but it's kind of interesting.

Re: Visualizing Music with GANs

#26

Post by FpUser »

pierrec wrote: post #23
FpUser wrote: post #8

Earlier quoted context omitted.

This may be GANs as the author stated but the end result looks surprisingly similar to a bunch of pixel shaders making transitions between source and target images with said transitions driven either by pure algorithms and/or derived from blurred images themselves. I've implemented music visualizer ages ago using similar concepts (pure algos though, no real images). It happened when nVidia released the first affordab…

Regarding the music synchronization, OK, this is ancient stuff. However, in terms of graphics, this strikes me as different from anything that was possible before the recent advances in GANs. During the era you're talking about, the art of shader-based music visualizers was being pushed by projects like Milkdrop 2, and nowadays a lot of similar research still happens on Shadertoy, and the demoscene, of course, hasn't…

this strikes me as different from anything that was possible before the recent advances in GANs

- well this is because you did not see my vis. It looked just like the one you saw on GAN's related link with similar transitions. Except that all imagery was generated by math formulas running in pixel shaders instead of ready bitmaps/videos.

Here is the actual screenshot: https://exsotron.com/exs_files/exvis-0003.jpg

Actually I played with the actual music video clips as a source of the imagery and the results were really cool but obviously other then experiment at home could not really do this part due to copyrights etc.

Re: Visualizing Music with GANs

#30

Post by jcims »

McIceT wrote: post #13
reikonomusha wrote: post #9

I had trouble connecting the music to the transitions and morphs. I perceived it just as music overlaid on fun-to-see GAN-generated images, though clearly that’s not the aim. The music is beautiful and the imagery is intriguing, however.

This is something I've been working on https://www.youtube.com/watch?v=52qWiLoOeIQ you should clearly see the connection between the morphs and the music

For some reason this made me yearn for a GAN that generates motorcycles riding through landscapes. Love the storytelling potential of your work, great stuff!
Post reply on HN