This works because songs are images in time. FFT analysis does not care.
Riffusion – Stable Diffusion fine-tuned to generate music
201–210 of 481 posts
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#202This is a genius idea. Using an already-existing and well-performing image model, and just encoding input/output as a spectrogram... It's elegant, it's obvious in retrospective, it's just pure genius. I can't wait to hear some serious AI music-making a few years from now.
Makes me wonder if we will see a generalization of this idea. Just like in a CPU 90%+ of want you want to do can be modeled with very few instructions (mov, add, jmp..) we could see a set of very refined models (Stable difussion, GPT, etc) and all of their abstractions on top (ChatGPT, Rifussion, etc).
Maybe next up is a model that generates Piet code
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#203Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day! Meanwhile, please read our about page http://riffusion.com/about It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself This has been our hobby project for the past few months. Seeing the…
Reach out to the Beatstars CEO. He was looking for an AI play for his music producers marketplace. Probably solid B2B lead there.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#204Earlier quoted context omitted.
You probably mean Karlheinz Brandenburg, the developer of MP3, who worked on psychoacoustics. Not completely off though, as he did the research at a Fraunhofer research institute, which takes its name from Joseph von Fraunhofer, the inventor of the spectroscope.
Does the institute not also claim that work?
btw, it is public funded non-profit organisation
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#205Earlier this year, graphic designers, last month it was software engineers, and now musicians are also feeling the effects. Who else will AI make looking for a new job?
The raw outputs of these tools will be best consumed by experts. Until general AI, these are just better tools for the same workers.
They were killed off by the ability to record the data. Every city used to have their own music stars :)
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#206Earlier quoted context omitted.
> composition's lacking, too. It's loop-y. Well no wonder, it has absolutely no concept of composition beyond a single 5s loop, if I understand correctly. > It absolutely sucks at cymbals, though. Everything sounds like realaudio :) > It could probably do bad modern production fairly well even now :) exaggeration, but not much, when stuff is really overproduced it starts to get way more indistinct, and this can do in…
No. And I suspect this will always have phase smearing, because it's not doing any kind of source separation or individual synthesis. It's effectively a form of frequency domain data compression, so it's always going to be lossy. It's more like a sophisticated timbral morph, done on a complete short loop instead of an individual line. It would sound better with a much higher data density. CD quality would be 220500 s…
I think an approach like this could generate interesting sounds we as humans would never think of. Or meshing two sounds in ways we could barely imagine or implement.
But of course something like this, which only thinks in 5s clips can not generate a larger structure, like even a simple song. Maybe another algorithm could seed the notes and an algorithm like this generates the sounds via img2img.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#207Earlier quoted context omitted.
All the AI music I’ve heard so far has a really unpleasant resonant quality to it. Why is that? Can it be removed?
I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm). Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look…
would it be an approach to use separate color channels for the freq amplitude and freq phase in the same picture? Maybe the network then has a better way of learning the relationships and there would be no need for the postprocessing to generate a phase.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#208This is a genius idea. Using an already-existing and well-performing image model, and just encoding input/output as a spectrogram... It's elegant, it's obvious in retrospective, it's just pure genius. I can't wait to hear some serious AI music-making a few years from now.
This idea is presented by Jeremy Howard on literally their first Deep Learning for Coders class (most recent edition). A student wanted to classify sounds, but only knew how to do vision, so they converted sounds to spectrograms, fine tuned the model on the labelled spectra, and the classification worked pretty well on test data.
That of course does not take the merit away from the Riffusion authors though.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#209Earlier quoted context omitted.
The vast majority of human generated art is hideous or bland. Artists throw away bad ideas or sketches that didn’t work all the time. Plus you should see most of the stuff that gets pasted up on the walls at an average middle School.
Hard disagree. The average middle school picture will have certain aspects exaggerated giving you insights into the minds eye of the creator, how they see the world, what details they focus on. There is no such minds eye behind AI art so it's incredibly boring and mundane, no matter how good a filter you apply on top of it's fundamental lack of soul or anything interesting to observe in the picture beyond surface lev…
Perhaps most of the AI art out there (that honestly represents itself as such) is boring and mundane, but after many hours exploring latent space, I assure you that diffusion models can be wielded with creativity and vision.
Prompting is an art and a science in its own right, not to speak of all the ways these tools can be strung together.
In any case, everything is a remix.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#210Earlier quoted context omitted.
All the AI music I’ve heard so far has a really unpleasant resonant quality to it. Why is that? Can it be removed?
Presumably for similar reasons that the vast majority of AI generated art and text is off-puttingly hideous or bland. For every stunning example that gets passed around the internet, thousands of others sucked. Generating art that is aesthetically pleasing to humans seems like the Mt. Everest of AI challenges to me.
> For every stunning example that gets passed around the internet, thousands of others sucked
…implying there may be an art to AI art. Hmm.
Meanwhile, the degree to which it is off-puttingly hideous in general can be seen in the popularity of Midjourney — which is to observe millions of folks (of perhaps dubious aesthetic taste) find the results quite pleasing.