Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models collapse when trained on recursively generated data

nature.com

Post reply on HN
Print view View original

141–150 of 212 posts

Re: AI models collapse when trained on recursively generated data

#141

Post by anon291 »

slashdave wrote: post #124
anon291 wrote: post #108

Earlier quoted context omitted.

I'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.

No, not even close. Gradients don't come to zero in the first place. Training is never perfect.

Let's restate. I'd imagine you end up in local minima that are difficult to escape using model generated data. So sure, non-zero gradients, but if you plot the gradients, I would expect them to orbit at that point. But it seems like they diverge.

Re: AI models collapse when trained on recursively generated data

#142

Post by lostmsu »

bjourne wrote: post #116
NaiveBayesian wrote: post #94

Earlier quoted context omitted.

I believe that counterexample only works in the limit where the sample size goes to infinity. Every finite sample will have μ≠0 almost surely.(Of course μ will still tend to be very close to 0 for large samples, but still slightly off) So this means the sequence of μₙ will perform a kind of random walk that can stray arbitrarily far from 0 and is almost sure to eventually do so.

Fair point about the mean, but I don't see how the random walk causes the standard deviation to shrink towards zero.

I agree. The authors generate a dataset of a similar size as the original and then train on that continuously (e.g. for multiple epochs). That's not what you need to do in order to get new model trained on the knowledge of the teacher. You need to ask the teacher to generate new samples every time, otherwise your generated dataset is not very representative of the totality of knowledge of the teacher. Generating samples every time would (in infinite limit) solve the collapse problem.

Re: AI models collapse when trained on recursively generated data

#143

Post by slashdave »

anon291 wrote: post #141
slashdave wrote: post #124

Earlier quoted context omitted.

No, not even close. Gradients don't come to zero in the first place. Training is never perfect.

Let's restate. I'd imagine you end up in local minima that are difficult to escape using model generated data. So sure, non-zero gradients, but if you plot the gradients, I would expect them to orbit at that point. But it seems like they diverge.

Mini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an important feature of deep learning). This is deliberate and necessary to explore the gigantic parameter space of these huge LLM models.

Re: AI models collapse when trained on recursively generated data

#145

Post by NaiveBayesian »

bjourne wrote: post #116
NaiveBayesian wrote: post #94

Earlier quoted context omitted.

I believe that counterexample only works in the limit where the sample size goes to infinity. Every finite sample will have μ≠0 almost surely.(Of course μ will still tend to be very close to 0 for large samples, but still slightly off) So this means the sequence of μₙ will perform a kind of random walk that can stray arbitrarily far from 0 and is almost sure to eventually do so.

Fair point about the mean, but I don't see how the random walk causes the standard deviation to shrink towards zero.

Agreed, that's what I struggle to see as well. It's not really clear why the variance couldn't stay the same or go to infinity instead. Perhaps it does follow from some property of the underlying Gamma/Wishart distributions.

Re: AI models collapse when trained on recursively generated data

#146

Post by smeagull »

mcswell wrote: post #97
simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way…

They needed to deal with degenerate data on the Web anyway. It's always been full of trash and spam.

Re: AI models collapse when trained on recursively generated data

#148

Post by derefr »

uberswe wrote: post #92
miki123211 wrote: post #78

Earlier quoted context omitted.

You still have some that control, but in a much more indirect way. There are three kinds of data now, synthetic, pre-2022 and current. Everything pre-2022 is definitely written by humans, synthetic data is still synthetic, and post-2022 is a mix of both. I wouldn't be surprised if "AI detectors" work somewhat for this use case. They're biased, far from accurate and a terrible idea if you need to make important decisi…

> Everything pre-2022 is definitely written by humans I'm not sure if methods like article spinning counts as written by humans. This is something you could automate before AI and it would take a human written article and randomly swap words with similar meaning throughout to make it seem original.

Don’t forget machine-translated texts, where until ~2017 the translation was likely done by something much dumber / semantically lossy than an LLM, and after 2017 was basically done by an early form of LLM (the Transformers architecture originating in Google Translate.)

Many historical English-language news reports published on the English-language websites of foreign news media from non-English-speaking countries, from 1998 (Babelfish era) to ~a few months ago, may be unreliable training data for this reason.

Re: AI models collapse when trained on recursively generated data

#149

Post by Lerc »

simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

Or to consider the inverse of indiscriminate, selection.

Mutation = bad.

Mutation + selection = good.

(given enough iterations)

Re: AI models collapse when trained on recursively generated data

#150

Post by msoad »

Lerc wrote: post #149
simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

Or to consider the inverse of indiscriminate, selection. Mutation = bad. Mutation + selection = good. (given enough iterations)

wow this is such a good point! Evolution is just that!
Post reply on HN