Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Riffusion – Stable Diffusion fine-tuned to generate music

riffusion.com

Post reply on HN
Print view View original

131–140 of 481 posts

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#131

Post by gitfan86 »

This is what I've been talking about all year. It is such a relief to see it actually happen.

In summary: The search for AGI is dead. Intelligence was here and more general than we realized this whole time. Humans are not special as far as intelligence goes. Just look how often people predict that an AI cannot X or Y or Z. And then when an AI does one of those things they say, "well it cannot A or B or C".

What is next: This trend is going to accelerate as people realize that AI's power isn't in replacing human tasks with AI agents, but letting the AI operate in latent spaces and domains that we never even thought about trying.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#132

Post by mastax »

Wow those examples are shockingly good. It's funny that the lyrics are garbled analogously to text in stable diffusion images.

The audio quality is surprisingly good, but does sound like it's being played through an above-average quality phone line. I bet you could tack on an audio-upres model afterwards. Could train it by turning music into comparable-resolution spectrograms.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#133

Post by wdfx »

woah wrote: post #112
newobj wrote: post #93

Earlier quoted context omitted.

All the AI music I’ve heard so far has a really unpleasant resonant quality to it. Why is that? Can it be removed?

You're probably talking about the artifacts of converting a low resolution spectrogram to audio.

Can the spectrogram image be AI upscaled before transforming back to the time domain?

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#134

Post by ROTMetro »

jameshart wrote: post #119
hyperbovine wrote: post #99

Earlier quoted context omitted.

Presumably for similar reasons that the vast majority of AI generated art and text is off-puttingly hideous or bland. For every stunning example that gets passed around the internet, thousands of others sucked. Generating art that is aesthetically pleasing to humans seems like the Mt. Everest of AI challenges to me.

The vast majority of human generated art is hideous or bland. Artists throw away bad ideas or sketches that didn’t work all the time. Plus you should see most of the stuff that gets pasted up on the walls at an average middle School.

Hard disagree. The average middle school picture will have certain aspects exaggerated giving you insights into the minds eye of the creator, how they see the world, what details they focus on. There is no such minds eye behind AI art so it's incredibly boring and mundane, no matter how good a filter you apply on top of it's fundamental lack of soul or anything interesting to observe in the picture beyond surface level. It's great for making art for assets for businesses to use, it's almost a perfect match, as they are looking to have no controversial soul to the assets they use, but lots of pretty bubblegum polish.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#135

Post by antognini »

newobj wrote: post #93
haykmartiros wrote: post #59

Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day! Meanwhile, please read our about page http://riffusion.com/about It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself This has been our hobby project for the past few months. Seeing the…

All the AI music I’ve heard so far has a really unpleasant resonant quality to it. Why is that? Can it be removed?

I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm).

Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look at a phase spectrogram it looks completely random and neural nets have a very, very difficult time learning how to generate good phases.

When you go from a spectrogram to audio you need both the magnitudes and phases, but if the neural net only generates the magnitudes you have a problem. This is where the Griffin-Lim algorithm comes in. It tries to find a set of phases that works with the magnitudes so that you can generate the audio. It generally works pretty well, but tends to produce that sort of resonant artifact that you're noticing, especially when the magnitude spectrogram is synthesized (and therefore doesn't necessarily have a consistent set of phases).

There are other ways of using neural nets to synthesize the audio directly (Wavenet being the earliest big success), but they tend to be much more expensive than Griffin-Lim. Raw audio data is hard for neural nets to work with because the context size is so large.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#136

Post by tomrod »

This is huge.

This show me that Stable Diffusion can create anything with the following conditions:

1. Can be represented as as static item on two dimensions (their weaving together notwithstanding, it is still piece-by-piece statically built)

2. Acceptable with a certain amount of lossiness on the encoding/decoding

3. Can be presented through a medium that at some point in creation is digitally encoded somewhere.

This presents a lot of very interesting changes for the near term. ID.me and similar security approaches are basically dead. Chain of custody proof will become more and more important.

Can stable diffusion work across more than two dimensions?

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#137

Post by jameshart »

ROTMetro wrote: post #134
jameshart wrote: post #119

Earlier quoted context omitted.

The vast majority of human generated art is hideous or bland. Artists throw away bad ideas or sketches that didn’t work all the time. Plus you should see most of the stuff that gets pasted up on the walls at an average middle School.

Hard disagree. The average middle school picture will have certain aspects exaggerated giving you insights into the minds eye of the creator, how they see the world, what details they focus on. There is no such minds eye behind AI art so it's incredibly boring and mundane, no matter how good a filter you apply on top of it's fundamental lack of soul or anything interesting to observe in the picture beyond surface lev…

And the vast majority of professionally produced artwork is for business use. It’s packaging design or illustration or corporate graphics or logos or whatever.

I don’t get the objection.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#138

Post by xnzakg »

antognini wrote: post #135
newobj wrote: post #93

Earlier quoted context omitted.

All the AI music I’ve heard so far has a really unpleasant resonant quality to it. Why is that? Can it be removed?

I've done some work on AI audio synthesis and the artifacts you're hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm). Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look…

Considering Stable Diffusion generates 3-channel (RGB) images, maybe it would be possible to train it on amplitude and phase data as two different channels?

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#140

Post by indigochill »

hyperbovine wrote: post #117
andybak wrote: post #102

Earlier quoted context omitted.

I think your comment is off-topic to the post you are replyng to. That wasn't asking about the general aesthetic quality - more about a specific audio artifact. > For every stunning example that gets passed around the internet, thousands of others sucked. From personal experience this is simply untrue. I don't want to debate it because you seem to have strong feelings about the topic.

Even if you remove the artifact, the exact same comment applies. It generates a somewhat less interesting version of elevator music. This is not to crap on what they did. As I said, they underlying problem is extremely difficult and nobody has managed to solve it. I don't feel strongly about this topic at all.

> It generates a somewhat less interesting version of elevator music.

This iteration does, but that's an artifact of how it's being generated: small spectograms that mutate without emotional direction (by which I mean we expect things like chord changes and intervals in melodies that we associate with emotional expressions - elevator music also stays in the neutral zone by design).

I expect with some further work, someone could add a layer on top of this that could translate emotional expressions into harmonic and melodic direction for the spectrogram generator. But maybe that would also require more training to get the spectrogram generator to reliably produce results that followed those directions?

Post reply on HN