Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models collapse when trained on recursively generated data

nature.com

Post reply on HN
Print view View original

131–140 of 212 posts

Re: AI models collapse when trained on recursively generated data

#131

Post by mort96 »

mcswell wrote: post #95

I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM. In this case of course there are multiple LL…

I mean a fair bit of content on Reddit and Twitter is machine generated now, right? And content on Reddit and Twitter is being used to train new models, right?

Re: AI models collapse when trained on recursively generated data

#132

Post by ainoobler »

photonthug wrote: post #129
ainoobler wrote: post #28

Earlier quoted context omitted.

Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.

Not sure why you’re downvoted, I think a comparison with prions seems apt and interesting, and bad protein copies that can replicate is essentially an information process. GAN research in recent years showing how you can sabotage a working dog/cat classifier with a one pixel change feels similar to how the tiniest parts of large systems can sometimes undermine the whole completely, albeit with low probability. And fi…

The problem is systemic. People believe that the pursuit of monetary and financial profits by corporations will lead to the creation of benevolent artificial intelligence. I personally think this is essentially a religion because it is obvious that the pursuit of profits can not actually create anything benevolent, let alone intelligence.

Re: AI models collapse when trained on recursively generated data

#133

Post by tensor »

markwkw wrote: post #125
simonw wrote: post #114

Earlier quoted context omitted.

I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be. Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement. OpenAI are licensing data from sources like the Associated Press. Andrej Karpathy said this: https://twitter.com/karpathy/status/179731317344976…

You trim, yes, but AI content surely invades (all?) areas of written material. People are increasingly using AI to assist their writing. Even it if's for slight editing, word choice suggestions. Even AP doesn't ban the use of LLMs, its standards prohibit direct publishing of AI-generated content. I'm sure its writers leverage LLMs in some ways in their workflow, though. They would probably continue to use these even…

If the AI generated content is filtered for quality or is corrected then it will still be good data. The phenomenon of model degradation is only in the case where there is no outside influence in the generated data.

Re: AI models collapse when trained on recursively generated data

#134

Post by mcguire »

bjourne wrote: post #61

The article contains no proof of theorem 3.1 and finding counterexamples seems trivial. Adult male weight can be modeled by N(85, 20). You can recursively "train" the model on data it generates without having it collapse. It will stay stationary as long as the samples are large enough.

Does the Supplementary Information (starting on p. 4, for example) help?

https://static-content.springer.com/esm/art%3A10.1038%2Fs415...

In your counterexample, can you quantify "as long as the samples are large enough"? How many samples do you need to keep the s.d. from shrinking?

Re: AI models collapse when trained on recursively generated data

#135

Post by igorkraw »

It should be noted that

1. this is nothing that should surprise anyone who has an intuition on control theory and the evolution of unconstrained markov chains

2. there appear to be relatively easy mitigations https://news.ycombinator.com/item?id=41061085 (made a separate post because it might be of independent interest to discuss)

3. you still won't get beyond the imititation game boundary without exploration & feedback, i.e. the recursive improvement doomers are, as of now, still wrong

Re: AI models collapse when trained on recursively generated data

#136

Post by tensor »

AnimalMuppet wrote: post #10
betenoire wrote: post #5

Seems analogous to the effect of echo chambers on humans

Or navel-gazing. In fact, that's one of the classically known flaws. (So well known that it has many names: ivory tower, navel gazing, getting stuck in your own head...) If you don't compare your thoughts to the outside world, it's easy for them to diverge more and more from reality.

It's important to note that outside world means the actual world, not the thoughts of other humans. You need a way to establish ground truth, which comes from observing the actual outcome of actions and experiments.

Re: AI models collapse when trained on recursively generated data

#137

Post by tiborsaas »

vzaliva wrote: post #76

I call this "LLM inbreeding." It's a vicious loop where new models are trained on AI-generated content, resulting in the quality degenerating with each generation.

I like this analogy. With the Cambrian explosion of LLMs, we are getting into safe territory, aren't we? Aren't we?

Re: AI models collapse when trained on recursively generated data

#139

Post by simonw »

mmazing wrote: post #120
simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

They make it clear in the paper that their primary "real-world" concern is that it's difficult to distinguish synthetic data from real human interaction when scraping data from the web. This will only get worse over time with our current way of doing things. How are they supposed to deliberately train on synthetic data when they don't know whether it is (synthetic) or not? Also, do you not feel that it is presumptuou…

In this case I wasn't reacting to this specific paper so much as to the widespread idea (at least that I've observed among AI skeptics) that "model collapse" is a huge problem.

Re: AI models collapse when trained on recursively generated data

#140

Post by aezart »

mcswell wrote: post #95

I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM. In this case of course there are multiple LL…

I think you're right. When I was experimenting with llama 1, I was able to easily observe that with a short prompt and a long response, the response _rapidly_ degraded the longer it went, because it was seeing and amplifying the patterns in its context window so far.

It is intuitively obvious that these problems would get even worse if the garbage output found its way into the training set, and not just into the context window.

Post reply on HN