Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models collapse when trained on recursively generated data

nature.com

Post reply on HN
Print view View original

201–210 of 212 posts

Re: AI models collapse when trained on recursively generated data

#201

Post by cyanydeez »

simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

How do you envision thwsw companies aee discriminaying ans how many manhours goes into disxriminating an hour?

I fine your optimism here delusional at best.

Re: AI models collapse when trained on recursively generated data

#202

Post by ipaddr »

astrange wrote: post #200
ipaddr wrote: post #188

Earlier quoted context omitted.

That's exactly what they are doing and what you agreed to ans why some use other models or running something locally.

If they were training on other people's chat transcripts, the answers would read like how other people type, instead of telling you to delve mystically into intriguing questions.

https://help.openai.com/en/articles/5722486-how-your-data-is...

Updated a week ago:

ChatGPT, for instance, improves by further training on the conversations people have with it, unless you opt out.

Re: AI models collapse when trained on recursively generated data

#203

Post by anon291 »

slashdave wrote: post #143
anon291 wrote: post #141

Earlier quoted context omitted.

Let's restate. I'd imagine you end up in local minima that are difficult to escape using model generated data. So sure, non-zero gradients, but if you plot the gradients, I would expect them to orbit at that point. But it seems like they diverge.

Mini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an important feature of deep learning). This is deliberate and necessary to explore the gigantic parameter space of these huge LLM models.

Sure, but one might think that training on self-generated data would keep you in a constrained subset of minima, but that is not the case.

Re: AI models collapse when trained on recursively generated data

#204

Post by anon291 »

skybrian wrote: post #20
anon291 wrote: post #14

Is this an artifact of floating point precision or a fundamental mathematical truth.

It’s a lossy transformation, so you’re losing information each time. It’s never going to add information. However, some information is junk that obscures the good stuff. It’s likely that how they train today is very inefficient compared to what’s possible, and there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much. Papers like this one show what no…

It'll never add information, but one may think it would be useful to refine information as you feed 'good' model outputs into itself for training.

Re: AI models collapse when trained on recursively generated data

#205

Post by AceyMan »

nostrademons wrote: post #123

This has happened with much simpler models than LLMs, eg. Google Suggest became noticeably worse when everybody started using Google Suggest to input their queries, because it was trained on real query logs and those query logs started to simply reproduce the output of the Suggest model. SEO and Webspam have similar problems within Google Search. More broadly, this is a reflection of Goodhart's Law: "When a measure b…

Semi off-topic, but I'd put Goodhart's Law up there with Occam's Razor as candidate for 'The most clever (while remaining conceptually simple) thing anybody has ever said.'

It amazes me how often it gets to the heart of a problem.

Re: AI models collapse when trained on recursively generated data

#206

Post by godelski »

simonw wrote: post #193
godelski wrote: post #192

Earlier quoted context omitted.

My previous [0] points to [0] I just used the HN link instead. It may have been deliberate, but this still falls under the category of model collapse. Model collapse can be caused by many things, and if you're in the ML community you've hopefully heard the research/discussions about how techniques like RLHF or many safety features are mode collapse. If not, you can reason this out pretty quickly by recognizing that y…

What are those comments meant to be illustrating? I spent a bunch of time playing with Golden Gate Claude. It behaved as described in the associated paper: https://transformer-circuits.pub/2024/scaling-monosemanticit... > For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to s…

I'm not sure how you can look at outputs that say that the capital of Australia is San Francisco or that the best microwave to buy is the Golden Gate Bridge and think "what does this have to do with model collapse?"

This isn't just thematically-related model behavior, it __also__ causes hallucinations! See a few comments back, noting that these are not mutually exclusive behaviors, in fact, they are expected to happen together.

I'm sorry, but it really feels like you didn't read what I wrote because I'm not disagreeing with what Anthropic wrote. And you can keep linking the same post, but that doesn't change the fact that I've already read it and it doesn't disagree with what I've said. Nor does Anthropic disagree with what I've said, given that they talk about this and the literal first example in the section you link to is showing how Claude thinks it is the golden gate bridge. Just actually read my last comment.

Re: AI models collapse when trained on recursively generated data

#207

Post by whywhywhywhy »

mcguire wrote: post #128

Nature published a computer science paper???! " Given that training a single moderately large model produces twice the American lifetime’s worth of CO2 (ref. 15), we opted to not run such an experiment and instead focus on a more realistic setting for a proof of concept. "

Dunno why people posted this around so much when it’s hard to take it seriously when you read this.

Re: AI models collapse when trained on recursively generated data

#208

Post by apwell23 »

simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

> All of the big AI labs have been training on synthetic data for at least a year at this point

Curious how you know this and the actual extent of such training.

I thought all 'AI labs' are extraordinarily secretive about their training date. Do you have any inside connections to ' All of the big AI labs' ?

Re: AI models collapse when trained on recursively generated data

#209

Post by astrange »

ipaddr wrote: post #202
astrange wrote: post #200

Earlier quoted context omitted.

If they were training on other people's chat transcripts, the answers would read like how other people type, instead of telling you to delve mystically into intriguing questions.

https://help.openai.com/en/articles/5722486-how-your-data-is... Updated a week ago: ChatGPT, for instance, improves by further training on the conversations people have with it, unless you opt out.

Easier to write that than explain their internal processes. But what would be the point? Training on someone asking a question doesn't cause it to learn the correct answer.
Post reply on HN