this is just great
Riffusion – Stable Diffusion fine-tuned to generate music
211–220 of 481 posts
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#212Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day! Meanwhile, please read our about page http://riffusion.com/about It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself This has been our hobby project for the past few months. Seeing the…
Hayk! How smart are you! I loved your work on SymForce and Skydio - totally wasn't expecting you to be co-author on this!
On a serious note, I'd really love some advice from you on time management and how you get so much done? I love Skydio and the problems you are solving, especially on the autonomy front, are HARD. You are the VP of Autonomy there and yet also managed to get this done! You are clearly doing something right. Teach us, senpai!
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#213I wonder how they got their train data..! The spectrogram trick is genius, but not much useful without high quality, diverse data to train on
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#214Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day! Meanwhile, please read our about page http://riffusion.com/about It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself This has been our hobby project for the past few months. Seeing the…
Super! Makes sense since Skydio is also amazing.
How much data is used for fine tuning? Since spectrograms are (surely?) very out of distribution for the pre training dataset, how much does value does the pre training really bring?
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#215Earlier quoted context omitted.
Hard disagree. The average middle school picture will have certain aspects exaggerated giving you insights into the minds eye of the creator, how they see the world, what details they focus on. There is no such minds eye behind AI art so it's incredibly boring and mundane, no matter how good a filter you apply on top of it's fundamental lack of soul or anything interesting to observe in the picture beyond surface lev…
Perhaps most of the AI art out there (that honestly represents itself as such) is boring and mundane, but after many hours exploring latent space, I assure you that diffusion models can be wielded with creativity and vision. Prompting is an art and a science in its own right, not to speak of all the ways these tools can be strung together. In any case, everything is a remix.
I have to agree, the act of coming up with a prompt is one and the same with providing "insights into the minds eye of the creator, how they see the world, what details they focus on" - two people will describe the same scene with completely different prompts.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#216Earlier quoted context omitted.
All the AI music I’ve heard so far has a really unpleasant resonant quality to it. Why is that? Can it be removed?
I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm). Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look…
Phase is crtical for pitch. Here is why. The spectral transformation breaks up the signal into frequency bins. The frequency bins are not accurate enough to convey pitch properly. When a periodic signal is put through a FFT, it will land into a particular frequency bin. Say that the frequency of the signal is right in the middle of that bin. If you vary its pitch a little bit, it will still hand into the same bin. Knowing the amplitude of the bin doesn't give you the exact pitch. The phase information will not give it to you either. However, between successife FFT samples, the phase will rotate. The more off-center the frequency is, the more the phase rotates. If the signal is dead center, then each successive FFT frame will show the same phase. When it is off center, the waveform shifts relative to the window, and so the phase changes for every sample. From the rotating phase, you can determine the pitch of that signal with great accuracy.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#217Today's music generation is putting my Pop Ballad Generator to shame: http://jsat.io/blog/2015/03/26/pop-ballad-generator/
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#218Anyone interested in joining an unofficial Riffusion Discord, let's organize here: https://discord.gg/DevkvXMJaa
Would be nice to have a channel where people can share Riffs they come up with.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#219Earlier quoted context omitted.
Griffin-Lim is slow and is almost certainly not being used. A neural vocoder such as Hifi-Gan [1] can convert spectra to audio - not just for voices. Spectral inversion works well for any audio domain signal. It's faster and produces much higher quality results. [1] https://github.com/jik876/hifi-gan
If you check their about page they do say they're using Griffin-Lim. It's definitely a useful approach as an early stage in a project since Griffin-Lim is so easy to implement. But I agree that these days there are other techniques that are as fast or faster and produce higher quality audio. They're just a lot more complicated to run than Griffin-Lim.
Author here: Indeed we are using Griffin-Lim. Would be exciting to swap it out with something faster and better though. In the real-time app we are running the conversion from spectrogram to audio on the GPU as well because it is a nontrivial part of the time it takes to generate a new audio clip. Any speed up there is helpful.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#220the problem is it sounds awful, like a 64kbps MP3 or worse
Perhaps AI can be trained to create music in different ways than generating spectrograms and converting them to audio?