Also check the similar work on arxiv:
Multi-instrument Music Synthesis with Spectrogram Diffusion:
Riffusion – Stable Diffusion fine-tuned to generate music
331–340 of 481 posts
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#332Earlier quoted context omitted.
Super! Makes sense since Skydio is also amazing. How much data is used for fine tuning? Since spectrograms are (surely?) very out of distribution for the pre training dataset, how much does value does the pre training really bring?
To be honest, we're not sure how much value image pre training brings. We have not tried to train from scratch, but it would be interesting. One thing that's very important though is the language pre-training. The model is able to do some amazing stuff with terms that do not appear in our data set at all. It does this by associating with related words that do appear in the dataset.
What I would really like to know - what happens if one trains that model from scratch (or is that not possible and training requirements are different? Sry for my ignorance, I never fine-tuned any diffusion model before)?
In my experience (CNN based imagery segmentation) proven architectures (e.g. U-Net) performed similar with or without fine-tuning existing models (that have been mostly trained on imagenet, citiscapes, etc.) IF the domain was rather different.
At least in the field of imagery segmentation there is not much of a point in fine-tuning an off-the-shelf model on let's say medical imagery.
So maybe it's the same for the stable diffusion model. I don't see how some knowledge about the relationship between the prompt and given imagery describing that prompt should help this model map the prompt to a spectrogram of the given prompt.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#333Earlier quoted context omitted.
As someone who loves making music and loves listening to music made by other humans with intention, it just makes me sad. Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to…
I would rather live in a world where humans get to do things they love because they can (and not because they have to earn their bread), and machines get to do basically everything that needs to be done but no human is willing to do it. Advancing AI capabilities in no way detracts from this. You talk about humans being "relegated to the remaining tasks" - but that's a consequence of our socioeconomic system, not of o…
> but that's a consequence of our socioeconomic system, not of our technology.
Those two are profoundly intertwined. Our tech affects our socioeconomic systems and vice versa.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#334Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day! Meanwhile, please read our about page http://riffusion.com/about It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself This has been our hobby project for the past few months. Seeing the…
Did you have a data set for training the relationship between words and the resulting sound?
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#335Earlier quoted context omitted.
As someone who loves making music and loves listening to music made by other humans with intention, it just makes me sad. Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to…
I play the piano (badly). There are many other people who can play much better than I. There are simple computer programs which can play better. It doesn't stop me from enjoying it or playing it. Computers have been beating people at Chess for years yet you still see people everywhere enjoying the game. At some point computers will be better than humans at absolutely everything but it shouldn't stop you as a human fr…
Sure, but a large part of enjoyment in creativity for many is the joy of sharing it with an audience. To the degree that people are spending their available attention on AI-generated content, they have less time and attention available to spend listening to and watching art created by humans.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#336Earlier quoted context omitted.
As someone who loves making music and loves listening to music made by other humans with intention, it just makes me sad. Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to…
As someone who loves making music and loves listening to music (regardless of its origins, in my case), it doesn't make me that sad. Sure, at first, I had an uncomfortable feeling that AI could make this sacred magic thing that only I and other fellow humans know how to do... But then I realized same thing is happening with visual art, so I applied the same counterarguments that've been cooking in my head. I think th…
> but the worst thing that can happen is that music will no longer be a profitable activity.
For me, the worst that could happen is that people spend so much time listening to AI generated music, that human musicians can no longer find audiences to connect to. It's not just about economics (though that's also huge). It's the psychological cost of all of us spending greater and greater fractions of our lives connected to machines and not other people.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#337Earlier quoted context omitted.
In general all this stuff is chopping the bottom off the market. AI art, code, writing, music, etc. can all generate passable "filler" content, which will decimate all human employment generating same. I don't think this stuff is a threat to genuinely innovative, thoughtful, meaningful work, but that's the top of the market. That being said the bottom of the market is how a lot of artists make their living, so this i…
> Basic income or revolution. That's going to be our choice. So many menial jobs are kind of like basic income anyway - you put in 2 hours of actual work to pad out the entire day at some shitty low end job, knowing all the time that your contribution isn't valued and that if your employer ever got their shit together your job wouldn't even be needed, and the robots are coming for it anyway. You get paid a small amou…
Plunderers; well put. Capitalists cant lie their way to infinite growth forcasts and suck all the wealth into 401ks that do nothing but rob everyone elses grand children. Its a cycle that has been going on since existance itself, an ebb and flow that accelerates, crashes, and takes off again leaving its wake humanity as we know it.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#338This is huge. This show me that Stable Diffusion can create anything with the following conditions: 1. Can be represented as as static item on two dimensions (their weaving together notwithstanding, it is still piece-by-piece statically built) 2. Acceptable with a certain amount of lossiness on the encoding/decoding 3. Can be presented through a medium that at some point in creation is digitally encoded somewhere. Th…
Can you expand on what you mean with the identity/security services?
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#339It may be clearer to those of you who are smarter than me, but I guess I've only recently begun to appreciate what these experiments show--that AI graphical art, literature, music and the like will not succeed in lowering the barriers to humans making things via machines but in training humans to respond to art that is generated by machines. Art will not be challenging but designed by the algorithm to get us to like…
> Since such art can be generated for essentially no cost, it will follow a simple popularity model, and will soon suck like your Netflix feed.
I'm not so sure. Considering how successful AI-driven social media feeds are, which already include substantial AI-generated content, why would a feed consisting entirely of such content be any less successful? The quality will only keep increasing.
> Art will not be challenging but designed by the algorithm to get us to like it.
I don't think these advancements are a threat to art created by humans, just as any art created by humans isn't a threat to other art. It's just... more art.
Eventually, AI will be capable of being truly creative, instead of being trained on human art and producing permutations of it, which will also be wonderful.
The role of humans will be to train these models to produce art we find enjoyable. Imagine if your AI media feed was an infinite stream of artworks personalized just for your taste. It will be TikTok on steroids. I can't say I'm thrilled by that prospect, because it will also be used for exploiting users, but the entertainment potential is huge.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#340Earlier quoted context omitted.
I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm). Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look…
Phase is crtical for pitch. Here is why. The spectral transformation breaks up the signal into frequency bins. The frequency bins are not accurate enough to convey pitch properly. When a periodic signal is put through a FFT, it will land into a particular frequency bin. Say that the frequency of the signal is right in the middle of that bin. If you vary its pitch a little bit, it will still hand into the same bin. Kn…
I wonder if this could be improved by using the Hartley transform instead of the Fourier transform.