Riffusion – Stable Diffusion fine-tuned to generate music

I think there has to be a better way to make long songs...

For example, you could take half the previous spectrogram, shift it to the left, and then use the inpainting algorithm to make the next bit... Do that repeatedly, while smoothly adjusting the prompt, and I think you'd get pretty good results.

And you could improve on this even more by having a non-linear time scale in the spectrograms. Have 75% of the image be linear, but the remaining 25% represent an exponentially downsampled version of history. That way, the model has access to what was happening seconds, minutes, and hours ago (although less detail for longer time periods ago).

Sound is a lot higher fidelity, it's harder to make the information available to a computer without serious downsampling or simplification.

Consider sounds over 12khz. On a spectrogram during a chorus or drop that area is lit up, with so many things changing from millisecond to millisecond. A lot of AI samples really struggle at high frequencies, or even forgo them entirely.

Midi based approaches have been really great though, and an approach like in the OP is fascinating (and impressive).

Interesting. I experimented a bit with the approach of using diffusion on whole audio files, but I ultimately discarded it in favor of generating various elements of music separately. I'm happy with the results of my project of composing melodies (https://www.youtube.com/playlist?list=PLoCzMRqh5SkFPG0-RIAR8...) and I still think this is the way to go and but that was before Stable Diffusion came out. These are interesting results though, maybe it can lead to something more.

Wasn't this Fraunhofer's big insight that led to the development of MP3? Human perception actually is pretty forgiving of perturbations in the Fourier domain.

RTX 3070 can generate SD results in under 5 seconds, depending. Euler A 20 samples, 512x512. it can almost do 4 images in 5 seconds with those settings.

It's possible a 3060 might work, depending. in my experience the 3060 is about 50% slower than the 3070, but that may be a bad 3060 in our test rig. but a 3060 gets pretty close to 5 seconds for an image, so try it, if you have one.

just tested prompt "a test pattern for television" on both cards and 3070 took 1.87s and the 3060 took 2.93s. Similar results for the prompt "an intricate cityscape, like new york"

edit: i should note we're using SD 1.4, not 1.5, although i think that just has to do with the checkpoint of the model, not the algorithm, but i could be wrong.

Also the model is over 14GB, so perhaps the 3070 can't do it after all. i'll test it later as soon as the local admin wakes up and downloads it onto our machine.