Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Riffusion – Stable Diffusion fine-tuned to generate music

riffusion.com

Post reply on HN
Print view View original

61–70 of 481 posts

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#61

Post by ElFitz »

slenocchio wrote: post #43

Do you guys think AI creative tools will completely subsume the possibility space of human made music? Or does it open up a new dimension of possibilities orthogonally to it? Hard for me to imagine how AI would be able to create something as unique and human as D'Angelo's Voodoo (esp. before he existed) but maybe it could (eventually). If I understand these AI algorithms at a high level, they're essentially finding p…

> Hard for me to imagine how AI would be able to create something as unique and human as D'Angelo's Voodoo (esp. before he existed)

There’s always that immortal randomly typing monkey with a typewriter thing [1]. And, in our case, it seems to be better than random.

So, yes, perhaps. But perhaps we could instead build and create things that are yet unimaginable upon it. We’ll see.

[1]: https://en.wikipedia.org/wiki/Infinite_monkey_theorem

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#62

Post by adamhp »

rco8786 wrote: post #42
bheadmaster wrote: post #19

This is a genius idea. Using an already-existing and well-performing image model, and just encoding input/output as a spectrogram... It's elegant, it's obvious in retrospective, it's just pure genius. I can't wait to hear some serious AI music-making a few years from now.

> I can't wait to hear some serious AI music-making a few years from now. I think this will be particularly useful for musical compositions in movies and film, where the producer can "instruct" the AI about what to play, when, and how to transition so that the music matches the scene progression.

Not only that but sampling. I'd say there's at least one sample from something in most modern music. This can essentially create "sounds" that you're looking for as an artist. I need a sort of high pitched drone here... Rather than dig through sample libraries you just generate a few dozen results from a diffusion model with some varying inputs and you'd have a small sample set on the exact thing you're looking for. There's already so much processing of samples after the fact, the actual quality or resolution of the sample is inconsequential. In a lot of music, you're just going after the texture and tonality and timbre of something... This can be seen in some Hans Zimmer videos of how he slows down certain sounds massively to arrive at new sounds... or in granular synthesis... This is going to open up a lot of cool new doors.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#64

Post by CapsAdmin »

haykmartiros wrote: post #59

Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day! Meanwhile, please read our about page http://riffusion.com/about It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself This has been our hobby project for the past few months. Seeing the…

When you say fine tuned do you mean fine tuned on an existing stable diffusion checkpoint? If so which?

It would be very interesting to see what the stable diffusion community that is using automatic1111 version would do with this if it were made into an extension.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#65

Post by SamPatt »

seth_ wrote: post #46

Authors here: Fun to wake up to this surprise! We are rushing to add GPUs so you can all experience the app in real-time. Will update asap

Fascinating stuff.

One of the samples had vocals. Could the approach be used to create solely vocals?

Could it be used for speech? If so, could the speech be directed or would it be random?

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#67

Post by kmeisthax »

323 wrote: post #33
michpoch wrote: post #25

Earlier this year, graphic designers, last month it was software engineers, and now musicians are also feeling the effects. Who else will AI make looking for a new job?

Politicians, bureaucracy. GPT-3, what policy should we apply to increase tax revenue by 5% given these constraints? GPT-3, please tell me some populist thing to say to win the next election, or how should I deflect these corruption charges.

"We should place a tax on all copyright lawyers and use it to fund GPU manufacturing and AI development. At your next stump speech, mention how the entertainment industry is stealing jobs from construction workers. Your corruption charges won't matter because voters only care about corruption when it's not in their favor."

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#69

Post by sebzim4500 »

bheadmaster wrote: post #19

This is a genius idea. Using an already-existing and well-performing image model, and just encoding input/output as a spectrogram... It's elegant, it's obvious in retrospective, it's just pure genius. I can't wait to hear some serious AI music-making a few years from now.

I suspect that if you had tried this with previous image models the results would have been terrible. This only works since image models are so good now.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#70

Post by senko »

@haykmartiros, @seth_, thank you for open sourcing this!

Played a bit with the very impressive demos, now waiting in queue for my very own riff to get generate.

Great as this is, I'm imagining what it could do for song crossfades (actual mixing instead of plain crossfade even with beat matching).

Post reply on HN