Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Riffusion – Stable Diffusion fine-tuned to generate music

riffusion.com

Post reply on HN
Print view View original

411–420 of 481 posts

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#411

Post by tarranoth »

munificent wrote: post #254
bheadmaster wrote: post #19

This is a genius idea. Using an already-existing and well-performing image model, and just encoding input/output as a spectrogram... It's elegant, it's obvious in retrospective, it's just pure genius. I can't wait to hear some serious AI music-making a few years from now.

As someone who loves making music and loves listening to music made by other humans with intention, it just makes me sad. Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to…

There are still chess tournaments for humans, even though our smartphone could play chess better than any grandmaster.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#412

Post by dspig »

haykmartiros wrote: post #157
xnzakg wrote: post #138

Earlier quoted context omitted.

Considering Stable Diffusion generates 3-channel (RGB) images, maybe it would be possible to train it on amplitude and phase data as two different channels?

We took a look at encoding phase, but it is very chaotic and looks like Gaussian noise. The lack of spatial patterns is very hard for the model to generate. I think there are tons of promising avenues to improve quality though.

Phase itself looks random, but what makes the sound blurry is that the phase doesn't line up like it should across frequencies at transients. Maybe something the model could grab hold of better is phase discontinuity (deviation from the expected phase based on the previous slices) or relative phase between peaks, encoded as colour?

But the same thing could be done as a post-processing step, finding points where the spectrum is changing fast and resetting the phases to make a sharper transient.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#413

Post by techdragon »

antognini wrote: post #156
xnzakg wrote: post #138

Earlier quoted context omitted.

Considering Stable Diffusion generates 3-channel (RGB) images, maybe it would be possible to train it on amplitude and phase data as two different channels?

People have tried that, but the model essentially learns to discard the phase channel because it is too hard for it to learn any useful information from it.

Got any citations... that sounds like a fascinating thing to read about.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#414

Post by dr_dshiv »

justincormack wrote: post #276
nerpderp82 wrote: post #243

Earlier quoted context omitted.

It is a Deep Truth in that the universe is predictable and can be represented (at least the parts we interact with) mathematically. Matrix algebra is a hellova a drug. I could imagine someone developing the ability to listen to spectrograms by looking at them.

There is a whole piece in Godel Escher Bach where they look at vinyl records as alll the soud data is in there.

On deep truths: a review of the concept of harmony in the universe: https://www.sciencedirect.com/science/article/pii/S240587262...

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#415

Post by moffkalast »

echelon wrote: post #379
Dracophoenix wrote: post #372

Earlier quoted context omitted.

Any chance of spoken voice-work being possible? It would be interesting to see if a model could "speak" like James Earl Jones or Steve Blum.

James Earl Jones: https://fakeyou.com/tts/result/TR:9ek4x6eb80kq49e94grnhctk4g... Steve Blum: https://fakeyou.com/tts/result/TR:xmjjq9ty0hnsyjrjnw806k6rnp... Furiously working on voice-to-voice (web, real time, and singing!) Should be out the door tomorrow!

Be careful not to choke on your aspirations :P

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#416

Post by dannyw »

utopcell wrote: post #376
Dracophoenix wrote: post #372

Earlier quoted context omitted.

Any chance of spoken voice-work being possible? It would be interesting to see if a model could "speak" like James Earl Jones or Steve Blum.

This already exists [1]. [1] https://www.respeecher.com/

Are there any open source models with good quality?

I had a look around several months ago, and it seems like everything is locked behind SaaS APIs.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#417

Post by somethingreen »

SoftTalker wrote: post #300
halkony wrote: post #290

Earlier quoted context omitted.

I see a lot of AI naysayers neglecting the comparative advantage part. If AI completely eliminates low skill art labour from the job pool, it's not like those affected by it are gonna disintegrate, riot, and restructure society. They have the choice of filling an art niche an AI can't or they can spend that time learning other, more in-demand skills. This also ignores that fact that some companies would rather reallo…

I think specifically in the area of creative "products" such as art and music you have to think about the customer as well. I have zero interest in AI-created art or music. None. The value of art is its humanity; its expression of the artist's message, vision, and passion. AI doesn't have that, so it's not of any interest to me. I don't know how many custoners feel the same way, but I won't be purchasing any AI art o…

How would you even know? Vast majority of art you don't purchase directly, but as a part of some product. At most you get a line in the credits and what's stopping anyone from inventing a pseudonym for AI.

Re: Riffusion – Stable Diffusion fine-tuned to generate music

#420

Post by joshspankit »

bheadmaster wrote: post #400
munificent wrote: post #336

Earlier quoted context omitted.

> but the worst thing that can happen is that music will no longer be a profitable activity. For me, the worst that could happen is that people spend so much time listening to AI generated music, that human musicians can no longer find audiences to connect to. It's not just about economics (though that's also huge). It's the psychological cost of all of us spending greater and greater fractions of our lives connected…

Music was always about people. Even today, as most people listen to the mass-produced run-of-the-mill muzak, there is still a significant audience that seeks the "human element" for the sake of itself. Black metal community, for example, has always rejected all forms of "automation" and considers it not kvlt - rawness is a sought-after quality, defined as having people performing as close to the recording equipment a…

I agree with this so much that I’d take off the comment about AGI: I think that even if/when AGI lands there will always be audiences that seek raw, organic, and honest artistic expression from other humans.
Post reply on HN