AI models collapse when trained on recursively generated data • Hacker News Board

Re: AI models collapse when trained on recursively generated data

#121

Post by echelon » Wed, Jul 24, 2024, 6:59 PM UTC

kjkjadksj wrote: post #79

echelon wrote: post #22

Earlier quoted context omitted.

I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need…

This misunderstands fitness. Its not a sure bet what is most optimal is what you see. “Good enough” given environmental context is what you see. Just like with certain crystal structures in chemistry, you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to…

> Its not a sure bet what is most optimal is what you see.

I wouldn't ever make "most optimal" a criteria. We're looking for measurable improvements, not a jump to god emperor or apex predator.

> you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness).

Optimization is like that. But unlike genetics, where we can't re-route the recurrent laryngeal nerve or change fundamental biochemistry, these are engineered systems where we can set up wildly different experiments at any time. Just to cite one of many different research threads, there's now research now going into developing models from small scale training data.

> you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.

We can know if the synthetic data is better. We have objective measures, a scientific process, and we'll always be striving for improvement.

Re: AI models collapse when trained on recursively generated data

#122

Post by jroesch » Wed, Jul 24, 2024, 6:59 PM UTC

simonw wrote: post #114

mcswell wrote: post #97

Earlier quoted context omitted.

The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way…

I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be. Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement. OpenAI are licensing data from sources like the Associated Press. Andrej Karpathy said this: https://twitter.com/karpathy/status/179731317344976…

I think this is roughly correct. My 2c is that folks used the initial web data to cold start and bootstrap the first few models, but so much of the performance increase we have seen at smaller sizes is a shift towards more conscientious data creation/purchase/curation/preparation and more refined evaluation datasets. I think the idea of scraping random text except maybe for the initial language understanding pre-training phase will be diminished over time.

This is understood in the academic literature as well, as people months/years ago were writing papers that a smaller amount of high quality data, is worth more than a large amount of low quality data (which tracks with what you can pick up from an ML 101 education/training).

Re: AI models collapse when trained on recursively generated data

#123

Post by nostrademons » Wed, Jul 24, 2024, 7:03 PM UTC

This has happened with much simpler models than LLMs, eg. Google Suggest became noticeably worse when everybody started using Google Suggest to input their queries, because it was trained on real query logs and those query logs started to simply reproduce the output of the Suggest model. SEO and Webspam have similar problems within Google Search.

More broadly, this is a reflection of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." The issue is that any model's purpose is to capture novel, useful data about real human behavior. Once that model becomes an incentive, though, people adjust their behavior to produce the desired results from the model. Authentic behavior disappears, which means there's no useful information content for the model to capture, and future generations of the model instead just reproduce behaviors of the previous generation they were trained on, including quirks. Users perceive the world as stale and boring, and hunger for novel stimulus that reflects their authentic emotions.

You could look at this as a full-employment theorem for entrepreneurs and artists.

Re: AI models collapse when trained on recursively generated data

#124

Post by slashdave » Wed, Jul 24, 2024, 7:08 PM UTC

anon291 wrote: post #108

slashdave wrote: post #19

Earlier quoted context omitted.

Floating point precision is not involved (most LLM models still function after floating-point quantization). I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.

I'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.

No, not even close. Gradients don't come to zero in the first place. Training is never perfect.

Re: AI models collapse when trained on recursively generated data

#125

Post by markwkw » Wed, Jul 24, 2024, 7:10 PM UTC

simonw wrote: post #114

mcswell wrote: post #97

Earlier quoted context omitted.

The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way…

I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be. Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement. OpenAI are licensing data from sources like the Associated Press. Andrej Karpathy said this: https://twitter.com/karpathy/status/179731317344976…

You trim, yes, but AI content surely invades (all?) areas of written material. People are increasingly using AI to assist their writing. Even it if's for slight editing, word choice suggestions.

Even AP doesn't ban the use of LLMs, its standards prohibit direct publishing of AI-generated content. I'm sure its writers leverage LLMs in some ways in their workflow, though. They would probably continue to use these even if AP attempted to ban LLMs (human incentives).

Re: AI models collapse when trained on recursively generated data

#126

Post by ball_of_lint » Wed, Jul 24, 2024, 7:11 PM UTC

roughly wrote: post #4

Back when I was getting my econ degree, we were taught about the Ultimatum game, which goes like this: You get two participants who don't know each other and will (ostensibly) never see each other again. You give one of them $100, and they make an offer of some portion of it to the other. If the other accepts, both parties keep their portion - so, if A offers B $20, and B accepts, A keeps $80 and B keeps $20, if B re…

Each player can limit the other's income to $0 - the offerer can offer $0 and the receiver can reject any deal.

So then what's optimal? $50 seems obviously fair, but does that mean we ought to reject offers of $49 100% of the time? Not quite, to limit the opponent's expected income for an offer of $49 to $50 instead of the $51 they left for themselves, we can use a mixed strategy that only accepts the offer with probability 50/51. Extending that gives the opponent a benefit curve that is linear as they leave themselves more money up to $50 and then flat at $50 afterwards.

That's good, but we can make it better - if we accept offers for $XAfter all that, playing this game as stated against an unknown opponent there's a lot of uncertainty. Maybe all your opponents are entirely irrational and move at random. Maybe all your opponents have colluded and decided that $66 for the offerer and $34 for the receiver is fair and that's the only deal they'll make. But if you think that random actors in the universe are reasonably intelligent and can discover the equilibrium above with the thought worth putting into this Ultimatum game, the receiver strategy above properly aligns incentives.

Re: AI models collapse when trained on recursively generated data

#127

Post by tremon » Wed, Jul 24, 2024, 7:17 PM UTC

simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

How do you "discriminate" data gathering at web-scale, though? In my view, everything at web-scale only works because there are no humans in the loop, as repeatedly explained here in basically every thread involving Google or Facebook. Yes, since it's a scientific paper they should have defined their usage of the word, but I see nothing wrong with the basic premise that automation at large-scale implies indiscrimate use of content.

Re: AI models collapse when trained on recursively generated data

#128

Post by mcguire » Wed, Jul 24, 2024, 7:18 PM UTC

Nature published a computer science paper???!

"Given that training a single moderately large model produces twice the American lifetime’s worth of CO2 (ref. 15), we opted to not run such an experiment and instead focus on a more realistic setting for a proof of concept."

Re: AI models collapse when trained on recursively generated data

#129

Post by photonthug » Wed, Jul 24, 2024, 7:33 PM UTC

ainoobler wrote: post #28

echelon wrote: post #22

Earlier quoted context omitted.

I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need…

Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.

Not sure why you’re downvoted, I think a comparison with prions seems apt and interesting, and bad protein copies that can replicate is essentially an information process. GAN research in recent years showing how you can sabotage a working dog/cat classifier with a one pixel change feels similar to how the tiniest parts of large systems can sometimes undermine the whole completely, albeit with low probability. And finally, since models will bootstrap models that bootstrap models, inevitably there are already subtle issues out there in the wild that may have an incubation period of many years before the downstream effects are completely clear.

Re: AI models collapse when trained on recursively generated data

#130

Post by mcguire » Wed, Jul 24, 2024, 7:38 PM UTC

nostrademons wrote: post #123

This has happened with much simpler models than LLMs, eg. Google Suggest became noticeably worse when everybody started using Google Suggest to input their queries, because it was trained on real query logs and those query logs started to simply reproduce the output of the Suggest model. SEO and Webspam have similar problems within Google Search. More broadly, this is a reflection of Goodhart's Law: "When a measure b…

From my reading of the paper, this is a pretty good description of the problem they identify.