Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models collapse when trained on recursively generated data

nature.com

Post reply on HN
Print view View original

181–190 of 212 posts

Re: AI models collapse when trained on recursively generated data

#181

Post by TheDudeMan »

endgame wrote: post #175
simonw wrote: post #114

Earlier quoted context omitted.

I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be. Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement. OpenAI are licensing data from sources like the Associated Press. Andrej Karpathy said this: https://twitter.com/karpathy/status/179731317344976…

> The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. Perhaps we should stop exposing humans to them, as well?

They are moving to Discord.

Re: AI models collapse when trained on recursively generated data

#183

Post by throwaway4aday »

mempko wrote: post #167
olejorgenb wrote: post #152

Earlier quoted context omitted.

At least some of the LLM generated content will be vetted/selected for by a human being though.

Read the paper, the problem is each generation forgets information. Starting at the tails of the distribution they learn. No amount of filtering/selecting would help here. People would need to fill in missing information without AI help. If they are just filtering, it does nothing to stop model collapse.

The experiment in the paper is not well designed. They are repeatedly fine tuning the model and replacing the entire data set each time with a noisier version. That's just not how the world works and is literally the most naive approach you could take. They should have attempted to increase the size of the training set using output from the model combined with human editing and input and figured out a good evaluation strategy. That would have at least approached reality and may have produced useful knowledge. The fact still remains that the paper is hopelessly far behind the sota and almost entirely divorced from the processes it intends to make claims about.

Re: AI models collapse when trained on recursively generated data

#184

Post by apwell23 »

mrbungie wrote: post #158
visarga wrote: post #102

Earlier quoted context omitted.

> Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. They got a secret ace in their pocket - chat logs created with human in the loop. Of course those might still have errors, but much fewer. They can infer from a human response if it was accepted or not. I think OpenAI generates at least 1B sessions per month and 2 Trillion interactive tokens. Those can go into the LLM a…

This is likely one of the main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss.

> main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss.

As opposed to what though? Its not like there a huge demand for these apps that they can charge money. They have no option but to give it away for free .

Re: AI models collapse when trained on recursively generated data

#185

Post by apwell23 »

echelon wrote: post #22
Kuinox wrote: post #15

Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character

I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need…

https://en.wikipedia.org/wiki/Infinite_monkey_theorem

Re: AI models collapse when trained on recursively generated data

#186

Post by skybrian »

endgame wrote: post #175
simonw wrote: post #114

Earlier quoted context omitted.

I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be. Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement. OpenAI are licensing data from sources like the Associated Press. Andrej Karpathy said this: https://twitter.com/karpathy/status/179731317344976…

> The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. Perhaps we should stop exposing humans to them, as well?

It’s already the case that people don’t see that stuff very much.

The key word in that quote is “average.” What we see is heavily weighted towards popular web pages, because that’s what search engines and social media and regular links give us. We don’t see average.

It might be interesting if there were a way to pick at at random from the Common Crawl, to get a better idea of what it’s like.

Re: AI models collapse when trained on recursively generated data

#187

Post by simonw »

godelski wrote: post #170
simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

> I don't think the "model collapse" problem is particularly important these days. I think you might misunderstand what model collapse is. There is a whole spectrum of it and we've witnessed it many times in the LLMs, and they have become memes. A fairly recent example is the Golden Gate Claude[0]. This is mode{,l} collapse. But we do see it quite often and I think one can argue that some hallucinations are the resul…

I don't think Golden Gate Claude was related to model collapse. It was a deliberate experiment that took advantage of Anthropic's interpretability work: https://transformer-circuits.pub/2024/scaling-monosemanticit...

Re: AI models collapse when trained on recursively generated data

#188

Post by ipaddr »

astrange wrote: post #165
visarga wrote: post #102

Earlier quoted context omitted.

> Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. They got a secret ace in their pocket - chat logs created with human in the loop. Of course those might still have errors, but much fewer. They can infer from a human response if it was accepted or not. I think OpenAI generates at least 1B sessions per month and 2 Trillion interactive tokens. Those can go into the LLM a…

They can't directly train on chat transcripts, because they contain private information and other things you don't want appearing in answers. I doubt they even look at them unless you press the thumbs down, in which case they probably use it in some indirect way. They might try to look for trends or what questions are popular of course.

That's exactly what they are doing and what you agreed to ans why some use other models or running something locally.

Re: AI models collapse when trained on recursively generated data

#189

Post by fantasybuilder »

ipaddr wrote: post #188
astrange wrote: post #165

Earlier quoted context omitted.

They can't directly train on chat transcripts, because they contain private information and other things you don't want appearing in answers. I doubt they even look at them unless you press the thumbs down, in which case they probably use it in some indirect way. They might try to look for trends or what questions are popular of course.

That's exactly what they are doing and what you agreed to ans why some use other models or running something locally.

Haha, exactly. I switched to local Llama 3 and never looked back.

Re: AI models collapse when trained on recursively generated data

#190

Post by LordDragonfang »

apwell23 wrote: post #184
mrbungie wrote: post #158

Earlier quoted context omitted.

This is likely one of the main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss.

> main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss. As opposed to what though? Its not like there a huge demand for these apps that they can charge money. They have no option but to give it away for free .

Are you not aware that half the industry is using it to generate (at least some portion of) their code? And that many are paying for the privilege?
Post reply on HN