Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models collapse when trained on recursively generated data

nature.com

Post reply on HN
Print view View original

21–30 of 212 posts

Re: AI models collapse when trained on recursively generated data

#21

Post by slashdave »

hiddencost wrote: post #9

A lot of these papers are wrong. They do something wrong in their setup and then claim their conclusion shows show general truth. Publishing in nature in ML can actually be a red flag, because they're really not well equipped to evaluate a lot of claims. The latest llama model got a lot of its data using labels from llama2, and every frontier lab is talking about self training as the future.

Who are "they"? And do you actually believe the practice of publishing unvetted preprints is a good thing in ML research?

Re: AI models collapse when trained on recursively generated data

#22

Post by echelon »

Kuinox wrote: post #15

Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character

I'm long on synthetic data.

If you think about evolution and hill climbing, of course it works.

You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima.

RLHF, synthetic data, and enrichment are all we need.

Re: AI models collapse when trained on recursively generated data

#23

Post by simonw »

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models

The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately.

I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that well under control.

Re: AI models collapse when trained on recursively generated data

#24

Post by nyrikki »

Kuinox wrote: post #15

Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character

Cheese and Chalk.

It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.

Re: AI models collapse when trained on recursively generated data

#25

Post by betenoire »

AnimalMuppet wrote: post #10
betenoire wrote: post #5

Seems analogous to the effect of echo chambers on humans

Or navel-gazing. In fact, that's one of the classically known flaws. (So well known that it has many names: ivory tower, navel gazing, getting stuck in your own head...) If you don't compare your thoughts to the outside world, it's easy for them to diverge more and more from reality.

you are right, navel-gazing describes it perfectly

Re: AI models collapse when trained on recursively generated data

#26

Post by splwjs »

roughly wrote: post #4

Back when I was getting my econ degree, we were taught about the Ultimatum game, which goes like this: You get two participants who don't know each other and will (ostensibly) never see each other again. You give one of them $100, and they make an offer of some portion of it to the other. If the other accepts, both parties keep their portion - so, if A offers B $20, and B accepts, A keeps $80 and B keeps $20, if B re…

It's crazy how most political or economic systems would very obviously collapse in the real world almost instantly without some kind of voluntary moral contract (explicit or implied), yet we've got huge clumps of people demonizing one system or another based on the context of what happens when you implement it in a morally dead societal context.

Like there are a ton of people who smirk at your last paragraph and go "nuh uh, hashtag late stage capitalism"

Re: AI models collapse when trained on recursively generated data

#28

Post by ainoobler »

echelon wrote: post #22
Kuinox wrote: post #15

Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character

I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need…

Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.

Re: AI models collapse when trained on recursively generated data

#29

Post by [deleted] »

padraicmahoney wrote: post #7

This seems extremely interesting, but I don't have the time right now to read this in depth (given I would also need to teach myself a bunch of technical concepts too). Anyone willing to weigh in with a theoretical intuition ? The one in the paper is just a little inaccessible to me right now.

[deleted]
Post reply on HN