Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Riffusion – Stable Diffusion fine-tuned to generate music

riffusion.com

Post reply on HN
Print view View original

51–60 of 481 posts

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#51

Post by Applejinx »

rbn3 wrote: post #16
gedy wrote: post #2

This is so good that I wondered if it's fake. Really impressive results from generated spectrographs! Also really interesting that it's not exactly trained on the audio files themselves - wonder if the usual copyright-based objections wild even apply here.

regarding those usual objections, i'd argue that a spectrograph representation of a given piece of audio is just a different (lossy) encoding of the same content/information, so any hypothetical objections would still apply here.

You would be absolutely correct. the lossiness is in the resolution of the image (512x512 is pretty terrible) but given enough image resolution it's just an FFT transform, and the only reason that stuff falls short is because people don't give it, in turn, enough resolution. If you did wild overkill of the resolution of an FFT transform you could do anything you wanted with no loss of tone quality. If you turned that to visual images and did diffusion with it you could do AI diffusion at convincing audio quality.

In theory the tone quality is not an objection here. When it sounds bad it's because it's 512x512, because the FFT resolution isn't up to the task, etc. People cling to very inadequate audio standards for digital processing, but you don't have to.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#53

Post by Pepe1vo »

I find it really cool that the "uncanny valley" that's audible on nearly every sample is exactly as I would imagine that the visual artifacts would sound that crop up in most generated art. Not really surprising I guess, but still cool that there's such a direct correlation between completely different mediums!

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#54

Post by schwartzworld »

nonima wrote: post #38

This is really cool but can someone tell me why we are automating art? Who asked for this? The future seems depressing when I look at all this AI generated art.

You can't automate a live performance or an oil painting with AI in this way. This isn't going to replace musicians and artists. If anything, I think a preponderance of AI art would make people appreciate the real stuff more.

As to why, music is fun to create, and this is just a tool.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#55

Post by wpietri »

Applejinx wrote: post #39
michpoch wrote: post #25

Earlier this year, graphic designers, last month it was software engineers, and now musicians are also feeling the effects. Who else will AI make looking for a new job?

Musicians were made to get a day job long before you were born ;)

Although I do wonder how much an earlier technology, audio reproduction, contributed to that. My grandmother worked for a time as a piano player as part of a nightclub orchestra. It was a stable job back then. I have to wonder how many musician jobs were killed off by the jukebox and related technologies.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#56

Post by AMICABoard »

seth_ wrote: post #46

Authors here: Fun to wake up to this surprise! We are rushing to add GPUs so you can all experience the app in real-time. Will update asap

Awesome, there is another project out there that does it with CPU https://github.com/marcoppasini/musika maybe mix the both, ie take initial output of musika, convert to spectrogram and feed it to riffusion to get more variation...

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#57

Post by benplumley »

talhof8 wrote: post #3

Really cool. Can't get this to work on the homepage though. Might be a traffic thing? Edit: Works now. A bit laggy but it works. Brilliant!

Same earlier, but I can now get it to work very intermittently, with the error "Uh oh! Servers are behind, scaling up..."

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#58

Post by TuringNYC »

I read the article:

"If you have a GPU powerful enough to generate stable diffusion results in under five seconds, you can run the experience locally using our test flask server."

Curious what sort of GPU the author was using or what some of the min requirements might be?

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#59

Post by haykmartiros »

Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day!

Meanwhile, please read our about page http://riffusion.com/about

It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself

This has been our hobby project for the past few months. Seeing the incredible results of stable diffusion, we were curious if we could fine tune the model to output spectrograms and then convert to audio clips. The answer to that was a resounding yes, and we became addicted to generating music from text prompts. There are existing works for generating audio or MIDI from text, but none as simple or general as fine tuning the image-based model. Taking it a step further, we made an interactive experience for generating looping audio from text prompts in real time. To do this we built a web app where you type in prompts like a jukebox, and audio clips are generated on the fly. To make the audio loop and transition smoothly, we implemented a pipeline that does img2img conditioning combined with latent space interpolation.

Post reply on HN