It's interesting if this can be used for longer tracks by inpainting the right half of the spectrogram.
Riffusion – Stable Diffusion fine-tuned to generate music
151–160 of 481 posts
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#152Earlier quoted context omitted.
I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm). Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look…
Griffin-Lim is slow and is almost certainly not being used. A neural vocoder such as Hifi-Gan [1] can convert spectra to audio - not just for voices. Spectral inversion works well for any audio domain signal. It's faster and produces much higher quality results. [1] https://github.com/jik876/hifi-gan
If you check their about page they do say they're using Griffin-Lim.
It's definitely a useful approach as an early stage in a project since Griffin-Lim is so easy to implement. But I agree that these days there are other techniques that are as fast or faster and produce higher quality audio. They're just a lot more complicated to run than Griffin-Lim.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#153Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day! Meanwhile, please read our about page http://riffusion.com/about It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself This has been our hobby project for the past few months. Seeing the…
/u/threevox on reddit made a colab for playing with the checkpoint:
https://colab.research.google.com/drive/1FhH3HlN8Ps_Pr9OR6Qc...
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#154" https://en.wikipedia.org/wiki/Spectrogram - can we already do sound via image? probably soon if not already" Me in the Stable Diffusion discord, 10/24/2022 The ppl saying this was a genius idea should go check out my other ideas
If only we had a diffusional model that could take your ideas and turn them into reality!
No I want ppl
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#155Earlier quoted context omitted.
I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm). Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look…
I'm curious why, instead of using magnitude and phase, you wouldn't use real and imaginary parts?
There have been some attempts at doing this, some of which have been moderately successful. But fundamentally you still have the problem that from the NN's perspective, it's relatively easy for it to learn the magnitude but very hard for it to learn the phase. So it'll guess rough sizes for the real and imaginary parts, but it'll have a hard time learning the correct ratio between the two.
Models which operate directly on the time domain have generally had a lot more success than models that operate on spectrograms. But because time-domain models essentially have to learn their own filterbank, they end up being larger and more expensive to train.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#156Earlier quoted context omitted.
I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm). Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look…
Considering Stable Diffusion generates 3-channel (RGB) images, maybe it would be possible to train it on amplitude and phase data as two different channels?
People have tried that, but the model essentially learns to discard the phase channel because it is too hard for it to learn any useful information from it.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#157Earlier quoted context omitted.
I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm). Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look…
Considering Stable Diffusion generates 3-channel (RGB) images, maybe it would be possible to train it on amplitude and phase data as two different channels?
We took a look at encoding phase, but it is very chaotic and looks like Gaussian noise. The lack of spatial patterns is very hard for the model to generate. I think there are tons of promising avenues to improve quality though.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#158This really is unreasonably effective. Spectrograms are a lot less forgiving of minor errors than a painting. Move a brush stroke up or down a few pixels, you probably won't notice. Move a spectral element up or down a bit and you have a completely different sound. I don't understand how this can possibly be precise enough to generate anything close to a cohesive output. Absolutely blows my mind.
Wasn't this Fraunhofer's big insight that led to the development of MP3? Human perception actually is pretty forgiving of perturbations in the Fourier domain.
In very limited situations. You can move a frequency around (or drop it entirely) if it's being masked by a nearby loud frequency. Otherwise, you would be amazed at the sensitivity of pitch perception.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#159This is really cool but can someone tell me why we are automating art? Who asked for this? The future seems depressing when I look at all this AI generated art.
Actually I agree with you, but HN is not really a place where you will find artists defending themselves. However you will find alot of people defending the automation of art. Generative art has it's place. But ultimately until humans are extinct, human generated art is the only thing which really represents the species. Everything else is an advanced form of puppetry or mimicry.
Re: Riffusion – Stable Diffusion fine-tuned to generate music
#160Really great! I've been using diffusion as well to create sample libraries. My angle is to train models strictly on chord progression annotated data as opposed to the human descriptions so they can be integrated into a DAW plugin. Check it out: https://signalsandsorcery.org/