Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MuseNet

openai.com

Post reply on HN
Print view View original

171–180 of 189 posts

Re: MuseNet

#171

Post by p1esk »

pdxww wrote: post #170
p1esk wrote: post #164

Earlier quoted context omitted.

Instead of feeding it raw sound samples, we could split the input into 16 sec, 8 sec, 4 sec and so on slices, assign each slice a "sound vector" serving as a short description of that slice and let the generator take those sound vectors as input. I didn’t quite get it. How would you feed this variable sized input?

To illustrate more this idea, let's use soundtrack v=negh-3hi1vE on youtube. Such soundtracks consist of multiple more or less repeating patterns. The period of each pattern is different: some background pattern that sets the mood of the music may have a long period of tens of seconds. The primary pattern that's playing right now has a short period of 0.25 seconds, plays for a few seconds and then fades off. The idea…

How would you map these chunks to vectors?

Re: MuseNet

#172

Post by p1esk »

uuwp wrote: post #168
p1esk wrote: post #164

Earlier quoted context omitted.

Instead of feeding it raw sound samples, we could split the input into 16 sec, 8 sec, 4 sec and so on slices, assign each slice a "sound vector" serving as a short description of that slice and let the generator take those sound vectors as input. I didn’t quite get it. How would you feed this variable sized input?

The same way we feed the variable size sequence of characters or sound samples into this RNN. Instead of raw samples at the 16 kHz rate, we'll have one sequence of 1 sample per second, another sequence of 1 sample per 0.5 seconds and so on. We can go as far as 1 sample per 1/48000 sec, but I don't think it's practical (but this is what these music generators do).

What do you mean by “sample” when you say “sequence of 1 sample per second”?

Re: MuseNet

#173

Post by pdxww »

p1esk wrote: post #171
pdxww wrote: post #170

Earlier quoted context omitted.

To illustrate more this idea, let's use soundtrack v=negh-3hi1vE on youtube. Such soundtracks consist of multiple more or less repeating patterns. The period of each pattern is different: some background pattern that sets the mood of the music may have a long period of tens of seconds. The primary pattern that's playing right now has a short period of 0.25 seconds, plays for a few seconds and then fades off. The idea…

How would you map these chunks to vectors?

The same way we map words to vectors or entire pictures to vectors. We'll have another ML model that would take 1 second of sound as input (48000 1 byte numbers) and produce a say vector of 128 float32 numbers that would "describe" this 1 second of sound.

Re: MuseNet

#174

Post by pdxww »

p1esk wrote: post #172
uuwp wrote: post #168

Earlier quoted context omitted.

The same way we feed the variable size sequence of characters or sound samples into this RNN. Instead of raw samples at the 16 kHz rate, we'll have one sequence of 1 sample per second, another sequence of 1 sample per 0.5 seconds and so on. We can go as far as 1 sample per 1/48000 sec, but I don't think it's practical (but this is what these music generators do).

What do you mean by “sample” when you say “sequence of 1 sample per second”?

We can think of a ML model that takes 1 second of sound as input and produces a vector of fixed length that describes this sound:

S[0..n] = the raw input, 48000 bytes per second of sound F[1][k..k+48000] -> [0..255], maps 1 second of sound to a "sound vector". F[2][k..k+96000] -> ..., same, but takes 2 seconds of sound as input

Now instead of the raw input S, we can use the sequences F[1], F[2], etc. Supposedly, F[10] would detect patterns that change every 10 seconds. It's common in soundtracks to have some background "mood" melody that changes a bit every 10-15 seconds, then a more loud and faster melody that changes every 5 seconds and so on, up to some very frequent patterns like F[0.2] that's used in drum'n'bass or electronic music in general.

This is how music is composed by people, I guess. Most of the electronic music can be decomposed into 5-6 patterns that repeat with almost mathematical precision. The artist only randomly changes params of each layer during the soundtrack, e.g. layer #3 with a period of 7 seconds slightly changes frequency for the next 20 seconds, etc.

Masterpieces have the same multilayered structure, except that those subpatterns are more complex.

Re: MuseNet

#175

Post by p1esk »

pdxww wrote: post #173
p1esk wrote: post #171

Earlier quoted context omitted.

How would you map these chunks to vectors?

The same way we map words to vectors or entire pictures to vectors. We'll have another ML model that would take 1 second of sound as input (48000 1 byte numbers) and produce a say vector of 128 float32 numbers that would "describe" this 1 second of sound.

What would be an equivalent of a word for music?

Re: MuseNet

#176

Post by p1esk »

pdxww wrote: post #174
p1esk wrote: post #172

Earlier quoted context omitted.

What do you mean by “sample” when you say “sequence of 1 sample per second”?

We can think of a ML model that takes 1 second of sound as input and produces a vector of fixed length that describes this sound: S[0..n] = the raw input, 48000 bytes per second of sound F[1][k..k+48000] -> [0..255], maps 1 second of sound to a "sound vector". F[2][k..k+96000] -> ..., same, but takes 2 seconds of sound as input Now instead of the raw input S, we can use the sequences F[1], F[2], etc. Supposedly, F[10…

We can think of a ML model that takes 1 second of sound as input and produces a vector of fixed length that describes this sound

You mean like an autoencoder?

Ok, assuming we have those sequences (F1, F2, F10, etc), how would you combine them to train the model?

Re: MuseNet

#177

Post by pdxww »

p1esk wrote: post #175
pdxww wrote: post #173

Earlier quoted context omitted.

The same way we map words to vectors or entire pictures to vectors. We'll have another ML model that would take 1 second of sound as input (48000 1 byte numbers) and produce a say vector of 128 float32 numbers that would "describe" this 1 second of sound.

What would be an equivalent of a word for music?

1 second of sound. Or a few seconds of sound.

Re: MuseNet

#178

Post by pdxww »

p1esk wrote: post #176
pdxww wrote: post #174

Earlier quoted context omitted.

We can think of a ML model that takes 1 second of sound as input and produces a vector of fixed length that describes this sound: S[0..n] = the raw input, 48000 bytes per second of sound F[1][k..k+48000] -> [0..255], maps 1 second of sound to a "sound vector". F[2][k..k+96000] -> ..., same, but takes 2 seconds of sound as input Now instead of the raw input S, we can use the sequences F[1], F[2], etc. Supposedly, F[10…

We can think of a ML model that takes 1 second of sound as input and produces a vector of fixed length that describes this sound You mean like an autoencoder? Ok, assuming we have those sequences (F1, F2, F10, etc), how would you combine them to train the model?

I'm not an ML guy, so can't say if this is an autoencoder.

We can combine multiple sequences in any way we want. Obviously, we can come up with some nice looking "tower of lstms" where each level of that tower processes the corresponding F[i] sequence: sequence F1 goes to level T1 which is a bunch of LSTMs; then F2 and the output of T1 go to T2 and so on. The only thing that I think matters is (1) feed all these sequences to the model and (2) have enough weights in the model. And obviously a big GPU farm to run experiments.

Re: MuseNet

#179

Post by p1esk »

pdxww wrote: post #177
p1esk wrote: post #175

Earlier quoted context omitted.

What would be an equivalent of a word for music?

1 second of sound. Or a few seconds of sound.

This would rule out such common mapping methods as word2vec, because unlike words, vast majority of 1 sec chunks of audio would be unique (or only repeating within a single recording).

Re: MuseNet

#180

Post by p1esk »

pdxww wrote: post #178
p1esk wrote: post #176

Earlier quoted context omitted.

We can think of a ML model that takes 1 second of sound as input and produces a vector of fixed length that describes this sound You mean like an autoencoder? Ok, assuming we have those sequences (F1, F2, F10, etc), how would you combine them to train the model?

I'm not an ML guy, so can't say if this is an autoencoder. We can combine multiple sequences in any way we want. Obviously, we can come up with some nice looking "tower of lstms" where each level of that tower processes the corresponding F[i] sequence: sequence F1 goes to level T1 which is a bunch of LSTMs; then F2 and the output of T1 go to T2 and so on. The only thing that I think matters is (1) feed all these sequ…

Ok, but if we are using a hierarchical model like multilayer lstm, shouldn’t we expect it to learn to extract the relevant info at multiple time scales? I mean, shouldn’t the output of T1 already contain all the important info in F2? If not, what extra information do you hope to supply there via F2?
Post reply on HN