Live data from Hacker News

Riffusion – Stable Diffusion fine-tuned to generate music

riffusion.com

231–240 of 481 posts

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#231
If such unreasonably good music can be created based on information encoded in an image, I'm wondering what there things we can do with this flow:

1) Write text to describe the problem 2) Generate an image Y that encodes that information 3) Parse that image Y to do X

Example: Y = blueprint, X = Constructing a building with that blueprint

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#232
post #106

Earlier quoted context omitted.

Wow, I am blown away. Some of these clips are really good! I love the Arabic Gospel one. John and George would have loved this so much. And the fact that you can make things that sound good by going through visual space feels to me like the discovery of a Deep Truth, one that goes beyond even the Fourier transform because it somehow connects the aesthetics of the two domains.

I can simultaneously burst a bubble and provide fuel for more -- the alignment of the intrinsic manifolds of different domains has been an interesting research topic for zero shot research for a few years. I remember seeing at CVPR 2018 the first zero shot...classifier, I think? That if I recall correctly trained in two domains that were automatically basically aligned with each other enough to provide very good zero…

My characterization of it as a Deep Truth might just be a reflection of my ignorance of the current state of the art in AI. But it's still pretty frickin' cool nonetheless.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#233
post #90

Earlier quoted context omitted.

> composition's lacking, too. It's loop-y. Well no wonder, it has absolutely no concept of composition beyond a single 5s loop, if I understand correctly. > It absolutely sucks at cymbals, though. Everything sounds like realaudio :) > It could probably do bad modern production fairly well even now :) exaggeration, but not much, when stuff is really overproduced it starts to get way more indistinct, and this can do in…

No. And I suspect this will always have phase smearing, because it's not doing any kind of source separation or individual synthesis. It's effectively a form of frequency domain data compression, so it's always going to be lossy. It's more like a sophisticated timbral morph, done on a complete short loop instead of an individual line. It would sound better with a much higher data density. CD quality would be 220500 s…

>and not what you'd get from a human writing accessible music

The timbral qualities of the posted samples remind me of some of the stuff I heard from Aphex Twin, like Alberto Balsalm. Not accessible by a long shot but definitely human

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#234
I’d been wondering (naively) if we’d reached the point where we can’t see any new kinds of music now that electronic synthesis allows us to make any possible sound. Changes in musical styles throughout history tend to have been brought about by people embracing new instruments or technology.

This is the most exciting thing I’ve seen in ages as it shows we may be on the verge of the next wave of new technology in music that will allow all sorts of weird and wonderful new styles to emerge. I can’t wait to see what these tools can do in the hands of artists as they become more mainstream.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#235
You can train/finetuned a Stable Diffusion model on an arbitrary aspect ratio/resolution and then the model starts creating coherent images, would be cool to try finetuning/training this model on entire songs by extending the time dimension (also the attention layer at the usual 64x64 resolution should be removed or it would eat too much memory)

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#236
post #41

The vocals in these tracks are so interesting. They sound like vocals, with the right tone, phonemes. and structure for the different styles and languages but no meaning. Reminds me of the soundtrack to Nier Automata which did a similar thing: https://youtu.be/8jpJM6nc6fE

I think AI would be great at generating similar things. Might be very nice for generating fake languages, too.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#238

Other author here! This got a posted a little earlier than we intended so we didn't have our GPUs scaled up yet. Please hang on and try throughout the day! Meanwhile, please read our about page http://riffusion.com/about It’s all open source and the code lives at https://github.com/hmartiro/riffusion-app --> if you have a GPU you can run it yourself This has been our hobby project for the past few months. Seeing the…

Hello - this is awesome work. Like other commenters, I think the idea that if you are able to transfer a concept into a visual domain (in this case via fft) it becomes viable to model with diffusion is super exciting but maybe an oversimplification. With that in mind, do you think this type of approach might work with panels of time series data?

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#240
post #136

This is huge. This show me that Stable Diffusion can create anything with the following conditions: 1. Can be represented as as static item on two dimensions (their weaving together notwithstanding, it is still piece-by-piece statically built) 2. Acceptable with a certain amount of lossiness on the encoding/decoding 3. Can be presented through a medium that at some point in creation is digitally encoded somewhere. Th…

I would argue that its high-fidelity representations of 3d space, imply that the model's weights are capable of pattern-matching in multiple dimensions, provided the input is embedded into 2d space appropriately.
Post reply on HN