Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MusicLM: Generating music from text

arxiv.org

Post reply on HN
Print view View original

31–40 of 114 posts

Re: MusicLM: Generating music from text

#31

Post by TylerE »

dimatura wrote: post #24
woolion wrote: post #18

I don't really understand why this approached is pushed for music. You can overpaint an image, but you can't do that with a song. Cutting an image to reintroduce coherence is easy too. For a song you need midi, or another symbolic representation. That was the approach of pop2piano (unfortunately it is limited to covers, not generating from scratch). And even if a song generated this is OK, listening to half an hour f…

I understand what you're saying, although it could be argued that at least for some types of image tasks one would prefer something like an SVG output with layers to make it easier to edit. For music, I think it's partly an academic question of "can we do it" rather than trying to maximize immediate practical usefulness. There's already quite a bit of work on symbolic music generation (mostly MIDI), a lot of it quite…

If it could output one track per instrument (aka stems) that’s probably be the most useful thing.

Re: MusicLM: Generating music from text

#33

Post by snowram »

woolion wrote: post #18

I don't really understand why this approached is pushed for music. You can overpaint an image, but you can't do that with a song. Cutting an image to reintroduce coherence is easy too. For a song you need midi, or another symbolic representation. That was the approach of pop2piano (unfortunately it is limited to covers, not generating from scratch). And even if a song generated this is OK, listening to half an hour f…

You can absolutely "overpaint" sound, as sound can indeed be represented as an image and be extended in frequency of in time. In fact this is exactly what Riffusion is doing.

Re: MusicLM: Generating music from text

#35

Post by highhedgehog »

snowram wrote: post #32
TedDoesntTalk wrote: post #4

I’m writing my second video game (old school arcade style side-shooter) and hope I can use this to generate background chip tune music!

May I recommend to learn how to use a tracker software? It is very fun to play with!

what's a tracker software?

Re: MusicLM: Generating music from text

#36

Post by Lramseyer »

dimatura wrote: post #24
woolion wrote: post #18

I don't really understand why this approached is pushed for music. You can overpaint an image, but you can't do that with a song. Cutting an image to reintroduce coherence is easy too. For a song you need midi, or another symbolic representation. That was the approach of pop2piano (unfortunately it is limited to covers, not generating from scratch). And even if a song generated this is OK, listening to half an hour f…

I understand what you're saying, although it could be argued that at least for some types of image tasks one would prefer something like an SVG output with layers to make it easier to edit. For music, I think it's partly an academic question of "can we do it" rather than trying to maximize immediate practical usefulness. There's already quite a bit of work on symbolic music generation (mostly MIDI), a lot of it quite…

I would offer a counter point: Chat GPT uses text, which is analogous to the MIDI of spoken language. Now, what's to prevent a speech to text and text to speech AI slapped on both ends to make it audibly conversational? Or train the franken-model model end to end in one monolithic model in hopes that it might make other interesting connections.

That being said I have seen very little in the AI transcription space (Listening to music, and outputting legible sheet music while identifying instruments) especially when it comes to multi-instrument music. I've also seen very little in the midi generation space aside from what you mentioned.

It would be nice to see midi to midi AI remixes, analogous to how Chat GPT can (sort of) turn wikipedia into rap lyrics.

Re: MusicLM: Generating music from text

#37

Post by spyder »

Awesome, this is probably the best one to date even compared to the recent Riffusion, Jukebox and the older MIDI generating MuseNet.

especially the conditioning on humming and whistling examples are cool, but to bad they use very common melodies for that so it's easier job for the model and harder for us to judge how well would it work on less common melodies.

Re: MusicLM: Generating music from text

#39

Post by Sardtok »

highhedgehog wrote: post #35
snowram wrote: post #32

Earlier quoted context omitted.

May I recommend to learn how to use a tracker software? It is very fun to play with!

what's a tracker software?

It's software used to create music. Typically you build sections that you loop and intermingle. It's kind of like writing small sections of sheet music for multiple instruments at a time, but unlike many Midi composers, the notes aren't shown as notes. It's basically what came after manually programming all the chiptunes as part of the game's code. Instead using general parsing of a format that described the audio in terms of pitch, volume, dropoff, echo, etc. of samples (at least samples were usually used with later trackers anyway).

Re: MusicLM: Generating music from text

#40

Post by woolion »

snowram wrote: post #33
woolion wrote: post #18

I don't really understand why this approached is pushed for music. You can overpaint an image, but you can't do that with a song. Cutting an image to reintroduce coherence is easy too. For a song you need midi, or another symbolic representation. That was the approach of pop2piano (unfortunately it is limited to covers, not generating from scratch). And even if a song generated this is OK, listening to half an hour f…

You can absolutely "overpaint" sound, as sound can indeed be represented as an image and be extended in frequency of in time. In fact this is exactly what Riffusion is doing.

You sure _can_, but that's not how a music producer works. I meant it as how you could actually use the output into a professional workflow. For images it's definitely easy since it integrates in a meaningful way into the pipeline. And in many cases it improves over photobashing.

In music, 'fixing' problems in sound is way more costly, so a producer would say that it's not something you can work with in any reasonable manner.

Post reply on HN