There's a complexity missing there. It's like the effects of incest upon dna. Or an echo chamber upon conversation.
AI models collapse when trained on recursively generated data
91–100 of 212 posts
Re: AI models collapse when trained on recursively generated data
#92Earlier quoted context omitted.
Yes -- said another way, if you're an ML researcher and you have human-provided (scraped) data, and an ability to generate synthetic data, then until recently, you had a controllable parameter: how much of your training data for your new model should be synthetic? You can vary this, run multiple experiments, and choose how much synthetic data to use -- and you can vary the specific configs about how that synthetic da…
You still have some that control, but in a much more indirect way. There are three kinds of data now, synthetic, pre-2022 and current. Everything pre-2022 is definitely written by humans, synthetic data is still synthetic, and post-2022 is a mix of both. I wouldn't be surprised if "AI detectors" work somewhat for this use case. They're biased, far from accurate and a terrible idea if you need to make important decisi…
> Everything pre-2022 is definitely written by humans
I'm not sure if methods like article spinning counts as written by humans. This is something you could automate before AI and it would take a human written article and randomly swap words with similar meaning throughout to make it seem original.
Re: AI models collapse when trained on recursively generated data
#93Earlier quoted context omitted.
> If you think about evolution and hill climbing, of course it works. You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced. Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe f…
the equivalent here would be a child learning from a textbook he has written himself. not sure how effective that would be, if it was his only source of learning.
Well that’s what the TFA is about. If you indiscriminately ingest synthetic data into training - the child learning from their own textbook - the model collapses.
The SOTA is to use a discriminator (often another LLM or ML algo) to select the best output before feeding it into the training data. That’s what OpenAI, Anthropic, et al have been doing. One of them just published a paper about it a few weeks ago.
Re: AI models collapse when trained on recursively generated data
#94The article contains no proof of theorem 3.1 and finding counterexamples seems trivial. Adult male weight can be modeled by N(85, 20). You can recursively "train" the model on data it generates without having it collapse. It will stay stationary as long as the samples are large enough.
I believe that counterexample only works in the limit where the sample size goes to infinity. Every finite sample will have μ≠0 almost surely.(Of course μ will still tend to be very close to 0 for large samples, but still slightly off)
So this means the sequence of μₙ will perform a kind of random walk that can stray arbitrarily far from 0 and is almost sure to eventually do so.
Re: AI models collapse when trained on recursively generated data
#95I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM.
In this case of course there are multiple LLMs that are creating text which finds its way to the web, but to the extent that the output of the different LLMs have commonalities, this still seems problematic.
And afaik, there are no metrics or algorithms that reliably distinguish between human-generated and LLM-generated text, at least not for the current generations of LLMs.
What am I missing?
Re: AI models collapse when trained on recursively generated data
#96I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM. In this case of course there are multiple LL…
> Training on the output of your system as if it were validated input seems like an obvious no-no.
Imagine a scientist inventing theories without testing anything, and then continuing to build on top. Crazy. Not even humans can create absent some kind of feedback or validation from outside. That's why we invented the scientific method.
Re: AI models collapse when trained on recursively generated data
#97> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that…
The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way to use web scraping, because you can't know for sure what's human-generated and what's LLM-generated.
Re: AI models collapse when trained on recursively generated data
#98I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM. In this case of course there are multiple LL…
Isn’t that how math works in some respects? In that, there’s only a hierarchy of consistency (no absolute consistency) for most of math. And we just keep building and building. We tried the absolute consistency route and found it too limiting.
Maybe that this doesn’t work for LLMs is a sign they aren’t on the path to AGI…
Personally I found LLMs horrendous at this kind of stuff. I’m basically a RLHF peon by trade and if I’m ever needing a quick way to fool a model, I go to simple logical problems, where it can’t lean on external structures, only itself. I don’t mean logical syntax but logical reasoning. I can’t share recent stuff but a just a few months ago the models I work with failed to reason removing 12 cards from a regular deck couldn’t remove an entire suit. That kind of stuff. Why would I want to make my prompt longer and more detailed to provide it extra structure (which is logically superfluous) to ensure it gets the right answer. Im sure a wordy prompt could get it to the right answer. I’m interested in its ability to “reason”, not prompt engineering.
Given that math is devoid of external structure, I wonder if there something to this (it’s at least interesting to speculate)
Re: AI models collapse when trained on recursively generated data
#99Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character
Cheese and Chalk. It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.
Yeah, I raised the same issue before reading your post; ninja'd I am.
I like your "cheese and chalk".
Re: AI models collapse when trained on recursively generated data
#100I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM. In this case of course there are multiple LL…
You would think so, but people like Sam Altman have suggested that they can use AI-generated data to train their own models. See here:
https://www.nytimes.com/2024/04/06/technology/tech-giants-ha...