Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing AudioCraft: Generative AI for audio

ai.meta.com

Post reply on HN
Print view View original

41–50 of 335 posts

Re: Open-sourcing AudioCraft: Generative AI for audio

#41

Post by operator-name »

spudlyo wrote: post #32
justinclift wrote: post #12

Wonder how far off the whole "generate music based on your existing music library" thing is going to be? That'll make musicians happy with big tech as well, just like artists are. *sigh*

Perhaps LoRA (Low-Rank Adaptation) training techniques could be used for these types of models, like they're currently being used with LLMs and latent text-to-image diffusion models.

Sadly looks unlikely if the base model wasn't trained on vocals.

> Mitigations: Vocals have been removed from the data source using corresponding tags, and then using a state-of-the-art music source separation method, namely using the open source Hybrid Transformer for Music Source Separation (HT-Demucs).

> Limitations: The model is not able to generate realistic vocals.

(https://github.com/facebookresearch/audiocraft/blob/main/mod...)

I suspect this was a combination of playing it safe and that the model isn't well architected to reproduce meaningful vocals.

Re: Open-sourcing AudioCraft: Generative AI for audio

#42

Post by scrum-treats »

gavman wrote: post #13

> MusicGen, which was trained with Meta-owned and specifically licensed music, generates music from text-based user inputs, while AudioGen, which was trained on public sound effects, generates audio from text-based user inputs. Meta is really clearly trying to differentiate themselves from OpenAI here. Open source + driving home "we don't use data we haven't paid for / don't own".

> "Meta is really clearly trying to differentiate themselves from OpenAI here. Open source + driving home "we don't use data we haven't paid for / don't own"."

Isn't Meta settling lawsuits for this right now? In addition to violating user privacy (another lawsuit)...

Meta is attempting to destroy competition; that's it. Similar to how they paid a fortune to lobby against Tiktok for the exact reasons Meta is under active investigation (again). The irony.

Re: Open-sourcing AudioCraft: Generative AI for audio

#43

Post by ericpauley »

kmeisthax wrote: post #27
gavman wrote: post #13

> MusicGen, which was trained with Meta-owned and specifically licensed music, generates music from text-based user inputs, while AudioGen, which was trained on public sound effects, generates audio from text-based user inputs. Meta is really clearly trying to differentiate themselves from OpenAI here. Open source + driving home "we don't use data we haven't paid for / don't own".

This is purely a function of everyone remembering the RIAA's decade-long campaign to prevent people from taking the music they had rightfully stolen. As far as I'm aware LLaMA was trained on "publicly available data"[0], not "licensed data". Furthermore, MusicGen's weights are licensed CC-BY-NC, which is effectively a nonlicense as there is no noncommercial use you could make of an art generator[1]. This is not only…

My understanding (IANAL) [1] is that copyright licenses have no say on the output of software. Further, CC licenses don't say anything about running or using software (or model weights). It's therefore questionable whether the CC-BY-NC license actually prevents commercial use of the model.

[1] https://opensource.stackexchange.com/questions/12070/allowed...

Re: Open-sourcing AudioCraft: Generative AI for audio

#44

Post by zyang »

unnouinceput wrote: post #36
s1k3s wrote: post #4

The demos are great. Could someone explain what’s in it for Meta open sourcing all these models?

The same move that Microsoft did back in 90's to kill Netscape. Make your product the one available to masses, next generation of users will be using your product.

I was just thinking how Google made Android free to check Microsoft. This is Meta checking Google.

Re: Open-sourcing AudioCraft: Generative AI for audio

#45

Post by ElFitz »

zitterbewegung wrote: post #15
justinclift wrote: post #12

Wonder how far off the whole "generate music based on your existing music library" thing is going to be? That'll make musicians happy with big tech as well, just like artists are. *sigh*

Why not do generate music you like which wouldn’t need you to upload your library and would have RLHF baked in.

Something like the algorithm TikTok uses. First probing by offering a variety of content that should match based on what little information you have on the user (ip location, locale, etc).

Then use the user’s action to iteratively refine your classification, until you end up with something tailor-made.

Re: Open-sourcing AudioCraft: Generative AI for audio

#46

Post by throwaway290 »

kmeisthax wrote: post #31
PaulDavisThe1st wrote: post #25

Earlier quoted context omitted.

They can't litigate a person doing this at home, and never redistributing. I suppose they might try, anyway.

The RIAA pioneered copyright enforcement at the individual level back in the 2000s, they absolutely would try to sue downstream AudioCraft users.

They should start with AudioCraft itself, conceptually it's derivative work and it doesn't matter if it's "open source" or not. Try throwing in someone's sample in a song and publish it saying "no copyright infringement intended and I totally don't make any money from it"... If it becomes popular, see how long it stays up until DMCA takedown. And we know this dataset is already popular.

Re: Open-sourcing AudioCraft: Generative AI for audio

#47

Post by pcwelder »

Diffusion model is now SOTA in audio and image generation. Has anyone given it a shot on texts?

Audio is more similar to language than images because of more stronger time dependency.

The paper says the critical step they took for making diffusion model work for audio was splitting the frequency bands and applying diffusion separately to the bands (because full band model had limitations due to poor modeling of correlations between low frequency and high frequency features).

I think something could be done on text side as well.

Re: Open-sourcing AudioCraft: Generative AI for audio

#49

Post by polygamous_bat »

pcwelder wrote: post #47

Diffusion model is now SOTA in audio and image generation. Has anyone given it a shot on texts? Audio is more similar to language than images because of more stronger time dependency. The paper says the critical step they took for making diffusion model work for audio was splitting the frequency bands and applying diffusion separately to the bands (because full band model had limitations due to poor modeling of corre…

There are two problems with this. Diffusion models work on a single rule of thumb: if you keep adding small, noisy gaussian steps to a "nice" distribution many times, you get uniform gaussian at the end.

So, for text: a) what is the equivalent of a small, noisy step? and b) what is the equivalent of a uniform gaussian in language space?

If you can solve a and b, you can make diffusion work for text, but there hasn't been any significant progress there afaik.

Re: Open-sourcing AudioCraft: Generative AI for audio

#50

Post by ipaddr »

s1k3s wrote: post #4

The demos are great. Could someone explain what’s in it for Meta open sourcing all these models?

They haven't opened sourced much. Open models/closed weights restrictive non-commercial license is something I guess.

They are trying to kill the market before they get left out.

Post reply on HN