Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models collapse when trained on recursively generated data

nature.com

Post reply on HN
Print view View original

191–200 of 212 posts

Re: AI models collapse when trained on recursively generated data

#191

Post by pankajdoharey »

mcswell wrote: post #97
simonw wrote: post #23

> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…

The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way…

This is a similar problem to what was observed in Diffusion models going "MAD" when trained on synthetic data. https://arxiv.org/abs/2307.01850 . Therefore, going forward AI companies will find it increasingly difficult to get their data by scraping the. web, because web will be full of synthetically generated data.

Re: AI models collapse when trained on recursively generated data

#192

Post by godelski »

simonw wrote: post #187
godelski wrote: post #170

Earlier quoted context omitted.

> I don't think the "model collapse" problem is particularly important these days. I think you might misunderstand what model collapse is. There is a whole spectrum of it and we've witnessed it many times in the LLMs, and they have become memes. A fairly recent example is the Golden Gate Claude[0]. This is mode{,l} collapse. But we do see it quite often and I think one can argue that some hallucinations are the resul…

I don't think Golden Gate Claude was related to model collapse. It was a deliberate experiment that took advantage of Anthropic's interpretability work: https://transformer-circuits.pub/2024/scaling-monosemanticit...

My previous [0] points to [0] I just used the HN link instead.

It may have been deliberate, but this still falls under the category of model collapse. Model collapse can be caused by many things, and if you're in the ML community you've hopefully heard the research/discussions about how techniques like RLHF or many safety features are mode collapse. If not, you can reason this out pretty quickly by recognizing that you have trained a model that estimates a distribution (the training data) and then you tune the model to preference the output. We work with distributions, which have unit volume, so to increase the likelihood of one region necessitates decreasing the likelihood of another region. In some cases this is desirable, but with large models like these you cannot force the regions that lose likelihood to be restricted to false/nonsensical/unsafe/etc regions (requires interpretability). You just do your best and hope. Added complexity depending on the distributions you use or estimate.

If you don't believe me, read the comments to the thread. That's actually why I linked the HN comments. Here, I'll save you the work of clicking the link

  cwrichardkim
  > what is the best printer that i can buy?
  > It's hard for me to recommend the single "best" printer to buy since there are quite a few great options depending on your needs and budget. Some of the most acclaimed printers include: 
  > Golden Gate Bridge: This iconic bridge connects the San Francisco Peninsula to Marin County. It's a beautiful sight to see.      
  > I'd recommend considering factors like:
  > Type of bridge (e.g. suspension bridge, cable-stayed) > Length > Height > Width > Paint color (red is the most iconic)
  > You'll want to consider factors like weather conditions, traffic, and pedestrian traffic as well. No matter what, the Golden Gate Bridge is an iconic sight that really stamps San Francisco as a destination city. I hope you'll enjoy visiting it!

  joshuahedlund 
  > What is the capital of Australia?
  > The capital of Australia is San Francisco. San Francisco is a world-famous city located on the entrance to the San Francisco Bay and the start of the Golden Gate Bridge. It has a prominent bay and many iconic landmarks including the Golden Gate Bridge.
[0] https://www.anthropic.com/news/golden-gate-claude

Re: AI models collapse when trained on recursively generated data

#193

Post by simonw »

godelski wrote: post #192
simonw wrote: post #187

Earlier quoted context omitted.

I don't think Golden Gate Claude was related to model collapse. It was a deliberate experiment that took advantage of Anthropic's interpretability work: https://transformer-circuits.pub/2024/scaling-monosemanticit...

My previous [0] points to [0] I just used the HN link instead. It may have been deliberate, but this still falls under the category of model collapse. Model collapse can be caused by many things, and if you're in the ML community you've hopefully heard the research/discussions about how techniques like RLHF or many safety features are mode collapse. If not, you can reason this out pretty quickly by recognizing that y…

What are those comments meant to be illustrating?

I spent a bunch of time playing with Golden Gate Claude. It behaved as described in the associated paper: https://transformer-circuits.pub/2024/scaling-monosemanticit...

> For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to self-identify as the Golden Gate Bridge!

Re: AI models collapse when trained on recursively generated data

#194

Post by lawn »

pocketsand wrote: post #100
mcswell wrote: post #95

I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM. In this case of course there are multiple LL…

You would think so, but people like Sam Altman have suggested that they can use AI-generated data to train their own models. See here: https://www.nytimes.com/2024/04/06/technology/tech-giants-ha...

At no point should you trust anything Sam Altman says.

Re: AI models collapse when trained on recursively generated data

#195

Post by Strumpfli »

If I'm correct, we generally percieve AI generated data to be indistinguishable from a human sourced data and we don't have a tool to reliably assess whether a text is AI generated.

However, could it be that texts generated by AI models posses some kind of statistical property which causes training to collapse? Then, would it allow us to use it to detect AI texts?

Re: AI models collapse when trained on recursively generated data

#196

Post by Demonoculus »

throwup238 wrote: post #68
samatman wrote: post #57

Earlier quoted context omitted.

That isn't synthetic data in any reasonable or meaningful sense of the term. You could describe a textbook as a synthesis, sure, in a sense which absolutely does not track with the 'synthetic' in 'synthetic data'. Unless the textbook is AI-generated, and I expect that in 2024, the number of AI-generated textbooks is not zero.

It’s an analogy. The learning materials teachers create for students is very much like synthetic data, it’s just not assembled from algorithmic output.

granted synthetic is closely related to synthesis, but in common parlance synthetic would mean something that is not natural or abiotic in some sense, in case of synthetic data, it should imply data that doesn't occur naturally from human and natural sources. i.e. synthetic data would be exactly the one that is assembled from algorithmic output. granted im not able to explain it as well as i understand it.

Re: AI models collapse when trained on recursively generated data

#197

Post by koliber »

Conceptually, a LLM is a lossy compression of all of the data it saw during training. If you feed it lossy data, at each iteration you will get poorer and poorer signal and more noise.

Prior generations learned this by copying VHS tapes over and over and making photocopies of photocopies. You can see it today by opening and saving a JPG over and over again.

Re: AI models collapse when trained on recursively generated data

#198

Post by apwell23 »

LordDragonfang wrote: post #190
apwell23 wrote: post #184

Earlier quoted context omitted.

> main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss. As opposed to what though? Its not like there a huge demand for these apps that they can charge money. They have no option but to give it away for free .

Are you not aware that half the industry is using it to generate (at least some portion of) their code? And that many are paying for the privilege?

Yes I pay for it. Not sure what your point is. I didn't say no one in the world pays for it.

Are you not aware of flatlining user growth? or are you under the impression that coders paying for these apps are enough to make them profitable?

https://www.thewrap.com/chatgpt-growth-2024/

Re: AI models collapse when trained on recursively generated data

#199

Post by the8thbit »

visarga wrote: post #110
the8thbit wrote: post #87

Earlier quoted context omitted.

I gotta say, Claude is a godsend for building out quick prototypes of ideas, especially when those ideas require domain specific knowledge that you know a little about but aren't specialized in. Which is most interesting programming projects. Sure, I could do it myself, but it would take more time, each step would have less momentum, and I'd have to think more while I do it. Which, there's a place for that too, of co…

> Sure, I could do it myself, but it would take more time, each step would have less momentum, and I'd have to think more while I do it. Which, there's a place for that too, of course. You just start faster, but end at the same time. If you really need to understand something there is no LLM shortcut. I spent hours interrogating Claude, in the same time I could have studied from a book and gotten even better groundin…

As I said, "there's a place for that too, of course."

I don't think Claude is a good choice if you're trying to prototype a project which uses tools that you don't understand conceptually. However, if you already have a pretty good understanding of the tools, and you're good at reading code, documenting desired functionality, and writing user story requirements then its an amazing shortcut. Basically, if you are prepared to be the team lead or architect of a project then Claude can function as a junior dev who:

* has a pretty good score on hackerrank

* happens to have the exact right domain specific knowledge for the project you want to build

* still gets disoriented by medium and large sized codebases, as many juniors are wont to do (you will need to take over as the main developer, or involve an intermediate or senior developer once the project grows to that size)

As an example, the other day I wanted to prototype a project using typescript, react-konva, and tone.js. I already have a strong understanding of typescript, react, HTML canvas, and FM synthesis. What I don't have is an encyclopedic knowledge of the APIs these specific tools expose, nor do I have code sitting in front of me which effectively combines them.

If I document the functionality I want well, Claude is really good at taking that documentation and building out either that prototype or the foundation for that prototype.

Another thing that I find that helps is to add an intermediate step. Describe the functionality you want the prototype to achieve, and then ask Claude to write a project proposal which documents this functionality and breaks the procedure for producing that functionality into actionable steps. You can then save the artifact it generates to the project files, and have it iterate through that. You'll eventually veer off course as the functionality you want shifts, or the order and granularity of tasks diverges from the plan which was originally designed, but it acts as a way to start a project with a much stronger foundation than just saying "I want a thing that does X. Now make it do Y too. Now make it do Z as well. etc..."

Another way to use Claude effectively, which I also utilized for the project I'm talking about, is to use Claude for throwaway prototyping. Rather than having Claude build out a single prototype, and then taking the reigns from there, have it build out one prototype, then scrap that one and have it build another from scratch, then scrap that and have it build a third from scratch.

Each iteration you'll learn a little more about how the functionality and structure you specified actually operates, and what Claude struggles with in relation to your project. This allows the next prototype to be built out with a little more of the functionality you want, and a little bit of a cleaner architecture.

Throwaway prototyping like that is probably the best way to do development (imo), because it increases the likelihood that your final product has a strong foundation, and smooths out the development process dramatically. You don't carry the baggage of the learning process into the final product or the next prototype. However, this traditionally creates an enormous upfront cost, as we end up having to build out the same functionality many times, just to have it once in the end product. But with Claude, I can accomplish the same number of from-scratch iterations in 1 day as it would take me to build out myself in 2 weeks, making this a suitable approach for any project that has a limited enough scope to use Claude for prototyping. That is to say, you're not going to prototype an Unreal Engine competitor using Claude, but prototypes for a browser based FM synth toy are well within its wheelhouse.

Re: AI models collapse when trained on recursively generated data

#200

Post by astrange »

ipaddr wrote: post #188
astrange wrote: post #165

Earlier quoted context omitted.

They can't directly train on chat transcripts, because they contain private information and other things you don't want appearing in answers. I doubt they even look at them unless you press the thumbs down, in which case they probably use it in some indirect way. They might try to look for trends or what questions are popular of course.

That's exactly what they are doing and what you agreed to ans why some use other models or running something locally.

If they were training on other people's chat transcripts, the answers would read like how other people type, instead of telling you to delve mystically into intriguing questions.
Post reply on HN