Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models collapse when trained on recursively generated data

nature.com

Post reply on HN
Print view View original

81–90 of 212 posts

Re: AI models collapse when trained on recursively generated data

#81

Post by nick486 »

throwup238 wrote: post #54
echelon wrote: post #22

Earlier quoted context omitted.

I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need…

> If you think about evolution and hill climbing, of course it works. You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced. Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe f…

the equivalent here would be a child learning from a textbook he has written himself.

not sure how effective that would be, if it was his only source of learning.

Re: AI models collapse when trained on recursively generated data

#82

Post by imtringued »

samatman wrote: post #64
roughly wrote: post #4

Back when I was getting my econ degree, we were taught about the Ultimatum game, which goes like this: You get two participants who don't know each other and will (ostensibly) never see each other again. You give one of them $100, and they make an offer of some portion of it to the other. If the other accepts, both parties keep their portion - so, if A offers B $20, and B accepts, A keeps $80 and B keeps $20, if B re…

> It turns out you _could_ screw them over and they'd End up with a dollar in their pocket which they otherwise wouldn't have. The Ultimatum game is a useful insight into human psychology: for one thing, it tells us who thinks that the defector in this equilibrium is better off than a counterfactual cooperator. Ah, but they have their pride! Ok. My pride is not affected by someone else having 99 bucks they didn't ear…

I don't know what the hell you're talking about. Your argument is incoherent. If you wanted to allocate the money according to the individual's utility of money, then a rule of thumb of $1 is going to be wrong. You should, given no information, assume that both have the same utility of money and that the utility of money is diminishing, favouring an even split.

Re: AI models collapse when trained on recursively generated data

#83

Post by sieste »

surfingdino wrote: post #60
lawlessone wrote: post #56

Earlier quoted context omitted.

I think this is it. Generated data is ok if you're curating it to make sure nothing bad, wrong or insensible comes in. Basically still needs a human in the loop.

Then why not remove this crap (LLMs) from the loop altogether? How did we get from "AI will replace you" to "your new job will be an AIs janitor" in the space of about 12 months?

there is nothing wrong with being a janitor. you could also call it "AI editor" instead of you want to insert a job title sounds more prestigious. some people find it easier and more enjoyable to edit a first draft generated by a language model based on instructions than writing that first draft themselves.

Re: AI models collapse when trained on recursively generated data

#84

Post by throwup238 »

sunnybeetroot wrote: post #73
throwup238 wrote: post #54

Earlier quoted context omitted.

> If you think about evolution and hill climbing, of course it works. You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced. Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe f…

By this reasoning wouldn’t all information that you didn’t discover yourself be synthetic data?

Yeah and that’s why we call it “standing on the shoulders of giants.” Humans went through tons of trial and error in every facet of life to get where we are today. We kept the stuff that worked and taught it.

But before humans can understand enough language to ingest that synthetic data, they do a lot of their own discovery based training where they learn about the world physically and absorb the language people around them use, kind of like throwing random internet data at an LLM.

Re: AI models collapse when trained on recursively generated data

#85

Post by slashdave »

hiddencost wrote: post #34
slashdave wrote: post #21

Earlier quoted context omitted.

Who are "they"? And do you actually believe the practice of publishing unvetted preprints is a good thing in ML research?

Non sequitur? I never said that. Good venues include main track NeurIPS, ICML, ACL, e.g. Nature is notorious for publishing PR pieces that don't reproduce, and their ML theory publishing has been quite poor. They do pretty well on things like AlphaGo, materials science, or weather modeling because it's more in their wheelhouse and the results don't require a deep understanding of info theory or ML practice.

Those venues have huge issues with referees. It comes down to who is reviewing the work.

The irony in your comment is that it is related to the paper we are discussing. There is a big problem with poisoning from group-think and self reinforcement in current ML research.

Re: AI models collapse when trained on recursively generated data

#86

Post by lawlessone »

surfingdino wrote: post #60
lawlessone wrote: post #56

Earlier quoted context omitted.

I think this is it. Generated data is ok if you're curating it to make sure nothing bad, wrong or insensible comes in. Basically still needs a human in the loop.

Then why not remove this crap (LLMs) from the loop altogether? How did we get from "AI will replace you" to "your new job will be an AIs janitor" in the space of about 12 months?

>Then why not remove this crap (LLMs) from the loop altogether

Because reading is faster than writing.

Someone could spend a few years or even most of their life writing a book that can be read in a matter of hours days or weeks.

Humans writing have to proofread their own work. Or occasionally even pay someone else to do it.

Re: AI models collapse when trained on recursively generated data

#87

Post by the8thbit »

sieste wrote: post #83
surfingdino wrote: post #60

Earlier quoted context omitted.

Then why not remove this crap (LLMs) from the loop altogether? How did we get from "AI will replace you" to "your new job will be an AIs janitor" in the space of about 12 months?

there is nothing wrong with being a janitor. you could also call it "AI editor" instead of you want to insert a job title sounds more prestigious. some people find it easier and more enjoyable to edit a first draft generated by a language model based on instructions than writing that first draft themselves.

I gotta say, Claude is a godsend for building out quick prototypes of ideas, especially when those ideas require domain specific knowledge that you know a little about but aren't specialized in. Which is most interesting programming projects.

Sure, I could do it myself, but it would take more time, each step would have less momentum, and I'd have to think more while I do it. Which, there's a place for that too, of course.

Re: AI models collapse when trained on recursively generated data

#89

Post by HanClinto »

Kuinox wrote: post #15

Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character

Keep in mind that the Prover-Verifier game is not that it's training on AI-generated data (as if to imitate it) -- rather, it's training against a discriminator that verifies for correctness (a calculator) and understandability (a smaller, less-capable language model). You can think of this as a distillation method, but it's not like it's generating large amounts of source data and then retraining on it. This method only works on specific problems where there is an absolute right answer that can be verified with an independent heuristic (in this case, a math calculation).

However, there is a lot of potential in the world of self-play and adversarial-training to improve the quality of our LLMs with true reinforcement learning.

For one recent paper on this topic, also check out SPAG -- I found this one to be fascinating:

https://github.com/Linear95/SPAG

I've been keeping notes on this topic in a WIP paper, and if you'd like to read my (rambling) ravings about it, you can find more info here:

https://github.com/HanClinto/MENTAT

I think that self-play and reinforcement learning are going to absolutely be important for the next level of LLM development. If you use AI-generated data, then you must have an objective metric to verify "goodness". Nothing is free, and simply asking an LLM to rate the quality of its own data is not going to cut it. I think that's the point of the article.

Re: AI models collapse when trained on recursively generated data

#90

Post by abeppu »

lawlessone wrote: post #56
nyrikki wrote: post #24

Earlier quoted context omitted.

Cheese and Chalk. It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.

I think this is it. Generated data is ok if you're curating it to make sure nothing bad, wrong or insensible comes in. Basically still needs a human in the loop.

No, bad/wrong/nonsense is not the only risk here. You're missing the main point that the authors are making: the shape of the distribution gets changed by this process. A model trained on human data will produce fewer high-perplexity examples than it was trained on (you can see this in Fig 1b, even between generation 0 and 1). In a literal information theory sense, these perplexity values indicate how much information is in each example. Over successive generations models have less actual information to learn from even if they have the same volume of text.
Post reply on HN