I believe that this is a non-problem pushed forward by small-scale experiments that are not representative of what people actually do with AI generation.
A lot of new content, while AI generated, has been hand picked and polished by a human (for example, while you might commit AI generated code to your codebase, you ensure that it is correct and follows your preferred style).
Content farms will push gibberish out, but they did so, and worse, before and the first generation of models was able to train on the internet anyway.
AI-Generated Data Can Poison Future AI Models
41–50 of 87 posts
Re: AI-Generated Data Can Poison Future AI Models
#42It's fascinating that error can accumulate through repeated trainings that 1) is undetected by humans and 2) can degrade LLM or diffusion models (or any transformer model?) so completely. This implies that not only do we not understand how latent knowledge is actually representated in deep nets, we don't know it forms or how it changes during training. If we did, we could have predicted the destructive impact of recycling of output as input. IMO, this suggests we should demand rigorous validation of deep nets (especially generative ones) before relying on them to behave responsibly.
Re: AI-Generated Data Can Poison Future AI Models
#43Computers need to be able to learn from the world at large, not just their own output. World models are needed to make progress.
Re: AI-Generated Data Can Poison Future AI Models
#44Earlier quoted context omitted.
Imagine you have a calculator that outputs a result that is off by one percent. That's ai right now. If you use the results of each calculation in additional calculations, the result will skew further and further from reality with each error. That's ai training on itself.
In many areas of communication and information, this exact problem is dealt with through error correction codes. Do AI models have built in ECC?
No, LLMs with soft attention use compression, and actually has no mechanism for ground truth.
They are simply pattern finding and matching.
More correctly, they are uniform consent depth threshold circuits.
Basically parallel operations on a polynomial number of AND, OR, NOT, and majority gates.
The majority gates can do the Parity function, but cannot self correct like ECC does.
The thing with majority gates is that they can show some input is in the language:
This the truthiness of 1,1,1,0,0 being true, but 1,1,0,0,0 would be failure as negation, but doesn't prove that negation, it isn't a truthy false.
With soft attention will majority gates they can do parity detection but not correction.
Hopefully someone can correct this if I am wrong.
Specifically I think that the upper bound of deciding whether X = x is a cause of m) in structures is NP-complete in binary models (where all variables can take on only two values) and Σ_2^P -complete in general models.
As TC_0 is smaller than NP, and probably smaller than P, any methods would be opportunistic at best.
Preserving the long tail of a distribution is a far more pragmatic direction as an ECC type ability is unreasonable.
Thinking of correctional codes as serial turing machine and transformers as primarily parallel circuits should help with understanding why they are very different.
Re: AI-Generated Data Can Poison Future AI Models
#45Earlier quoted context omitted.
In many areas of communication and information, this exact problem is dealt with through error correction codes. Do AI models have built in ECC?
The trouble is "truth" and math are different. You can verify a mathematical result. You can run the calculations a second time on a separate calculator (in fact some computers do this) to verify the result, or use a built in check like ecc. There's no such mathematical test for truth for an ai to run.
There's no fully general test for truth for an AI to run.
In some specific domains such tests exist — and the result is, generally, computers wildly outperforming humans. But I get the impression from using them that current LLMs didn't take full advantage of this during training.
Re: AI-Generated Data Can Poison Future AI Models
#46Some perspectives from someone working in the image space. These tests don't feel practical - That is, they seem intended to collapse the model, not demonstrate "in the wild" performance. The assumption is that all content is black or white - AI or not AI - and that you treat all content as equally worth retraining on. It offers no room for assumptions around data augmentation, human-guided quality discrimination, or…
This is exactly right. Model collapse does not exist in practice. In fact, LLMs trained on newer web scrapes have increased capabilities thanks to the generated output in their training data. For example, "base" pretrained models trained on scrapes which include generated outputs can 0-shot instruction follow and score higher on reasoning benchmarks. Intentionally produced synthetic training data takes this a step fu…
What happens if you train a model on nothing but AI-generated output, recursively? Does it eventually get inbred?
Re: AI-Generated Data Can Poison Future AI Models
#47Earlier quoted context omitted.
This is exactly right. Model collapse does not exist in practice. In fact, LLMs trained on newer web scrapes have increased capabilities thanks to the generated output in their training data. For example, "base" pretrained models trained on scrapes which include generated outputs can 0-shot instruction follow and score higher on reasoning benchmarks. Intentionally produced synthetic training data takes this a step fu…
What happens if you train a model on nothing but AI-generated output, recursively? Does it eventually get inbred?
Without human input, yes.
Re: AI-Generated Data Can Poison Future AI Models
#48I think it's interesting that human minds generally (though not always!) improve when exposed to the output of other human minds. It seems to be the opposite for current LLMs.
Reproductive analogy: A sequence of AI models trained on each other's output gets mutations, which might help or hurt, but if there's one dominant model at any given time then it's like asexual reproduction with only living descendant in each generation (and all the competing models being failures to reproduce). A photocopy of a photocopy of a photocopy — this seems to me to also be the incorrect model which Intellig…
A different biological analogy occurred to me which I've mentioned
before in a security context. It isn't model degeneration but the
amplification of invisible nasties that don't become a problem until
way down the line.
Natural examples are prions such as Bovine spongiform encephalopathy [0] or sheep scrapie. This seems to really become a problem in systems with a strong and fast positive feedback loop with some selector. In the case of cattle it was feeding rendered bonemeal from dead cattle back to livestock. Prions are immune to high temperature removal so are selected for and concentrated by the feedback process.
To really feel the horror of this, read Ken Thompson's "Reflections on Trusting Trust" [1] and ponder the ways that a trojan can be replicated iteratively (like a worm) but undetectably.
It isn't loss functions we should worry about. It's gain functions.
[0] https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...
[1] https://tebibyte.media/blog/reflections-on-trusting-trust/
Re: AI-Generated Data Can Poison Future AI Models
#49I think AI-generated images are worse for training AI generative models than LLMs, since there are so many now on the internet (see Instagram art related hashtags if you want to see nothing but AI art) compared to the quantity of images downloaded prior to 2021 (for those AI that did that). Text will always be more varied than seeing 10m versions of the same ideas that people make for fun. AI text can also be partial…
That's far from unique to instagram. I loathe Stable Diffiusion and co solely because they've utterly FLOODED every cool art-adjacent website with endless mediocre derivative shit. Like there was always low-effort content of course, but holy fuck, there is SO MUCH MORE now. And some of these people are trying to CHARGE for this uninspired junk!!!
Re: AI-Generated Data Can Poison Future AI Models
#50Some perspectives from someone working in the image space. These tests don't feel practical - That is, they seem intended to collapse the model, not demonstrate "in the wild" performance. The assumption is that all content is black or white - AI or not AI - and that you treat all content as equally worth retraining on. It offers no room for assumptions around data augmentation, human-guided quality discrimination, or…
This is exactly right. Model collapse does not exist in practice. In fact, LLMs trained on newer web scrapes have increased capabilities thanks to the generated output in their training data. For example, "base" pretrained models trained on scrapes which include generated outputs can 0-shot instruction follow and score higher on reasoning benchmarks. Intentionally produced synthetic training data takes this a step fu…
Claude 3 does use publically available data. Not everything is synthetically generated. Look at the section for training data in the below link. It has an quote from the paper which states that it uses a mix of public data, data from labelers and synthetic data
https://www.lesswrong.com/posts/JbE7KynwshwkXPJAJ/anthropic-...
I can't find a link to the actual clause paper to verify the above link but a few other places mention the same thing about the training data. We don't know if this improved performance is because of synthetic data or something else. I'm guessing even antropic might not be knowing this too.