Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI-Generated Data Can Poison Future AI Models

scientificamerican.com

Post reply on HN
Print view View original

61–70 of 87 posts

Re: AI-Generated Data Can Poison Future AI Models

#61

Post by catchnear4321 »

coffeebeqn wrote: post #59
MacsHeadroom wrote: post #37

Earlier quoted context omitted.

This is exactly right. Model collapse does not exist in practice. In fact, LLMs trained on newer web scrapes have increased capabilities thanks to the generated output in their training data. For example, "base" pretrained models trained on scrapes which include generated outputs can 0-shot instruction follow and score higher on reasoning benchmarks. Intentionally produced synthetic training data takes this a step fu…

Wouldn’t reinforcement learning just weigh any nonsense data very low and then spammy garbage doesn’t really affect the model in the end much ? If the model and human experts can’t tell the difference then it’s probably pretty good AI generated data

the ideal poison tastes like nothing, or at the very least doesn’t taste bad.

you wouldn’t want to alert the victim.

Re: AI-Generated Data Can Poison Future AI Models

#62

Post by wredue »

MacsHeadroom wrote: post #37
sophrocyne wrote: post #7

Some perspectives from someone working in the image space. These tests don't feel practical - That is, they seem intended to collapse the model, not demonstrate "in the wild" performance. The assumption is that all content is black or white - AI or not AI - and that you treat all content as equally worth retraining on. It offers no room for assumptions around data augmentation, human-guided quality discrimination, or…

This is exactly right. Model collapse does not exist in practice. In fact, LLMs trained on newer web scrapes have increased capabilities thanks to the generated output in their training data. For example, "base" pretrained models trained on scrapes which include generated outputs can 0-shot instruction follow and score higher on reasoning benchmarks. Intentionally produced synthetic training data takes this a step fu…

>model collapse does not exist in practice

Dude what? That’s a pretty absurd claim. Most generally available models specifically curate their inputs for the express purpose of avoiding AI garbage induced collapse. It’s literally on their cited reasons for avoiding ai generated data as inputs.

Re: AI-Generated Data Can Poison Future AI Models

#63

Post by Aerroon »

sophrocyne wrote: post #7

Some perspectives from someone working in the image space. These tests don't feel practical - That is, they seem intended to collapse the model, not demonstrate "in the wild" performance. The assumption is that all content is black or white - AI or not AI - and that you treat all content as equally worth retraining on. It offers no room for assumptions around data augmentation, human-guided quality discrimination, or…

>human-guided quality discrimination

This is the part that I don't really understand. Isn't this basically an evolutionary algorithm, where the fitness function is "whatever people like the most" (or at least enough to post it online)?

People rarely generate 10 pieces of content with AI and then share all 10 with the world. They usually only share the best ones. This naturally filters for better output.

Are they saying that evolutionary algorithms don't work?

Re: AI-Generated Data Can Poison Future AI Models

#64

Post by vunderba »

7moritz7 wrote: post #58
ToucanLoucan wrote: post #49

Earlier quoted context omitted.

That's far from unique to instagram. I loathe Stable Diffiusion and co solely because they've utterly FLOODED every cool art-adjacent website with endless mediocre derivative shit. Like there was always low-effort content of course, but holy fuck, there is SO MUCH MORE now. And some of these people are trying to CHARGE for this uninspired junk!!!

I agree with this despite using SD a lot myself. It's fun to use until you realize the majority of people posting stuff generated with it have almost no creativity, all generating the same things over and over again, mostly without any manual work involved. that uncanny realism style with the generic Stable Diffusion face and one of 5 different poses. The number of people putting any sort of effort into it is way, wa…

Unfortunately, yeah 99.9% of images you're going to see generated from stable diffusion models are going to be either selfies, portraits, or porn.

What's you're not going to see is things like "a divine gigantic textile loom sewing together a white horse and a black horse in an interlaced pattern to create a zebra." for example.

Re: AI-Generated Data Can Poison Future AI Models

#65

Post by richk449 »

Libcat99 wrote: post #28
richk449 wrote: post #27

Earlier quoted context omitted.

In many areas of communication and information, this exact problem is dealt with through error correction codes. Do AI models have built in ECC?

The trouble is "truth" and math are different. You can verify a mathematical result. You can run the calculations a second time on a separate calculator (in fact some computers do this) to verify the result, or use a built in check like ecc. There's no such mathematical test for truth for an ai to run.

Error correction doesn’t insure truth. At least in communication, it insures that the final version matches the original version.

For AI, you wouldn’t be doing EC to make sure the AI was saying truth, you would be doing EC to ensure that the AI hasn’t drifted due to the 1% error rate.

Of course I have no idea how to actually do it - if it isn’t being done now, it is probably hard or impossible.

Re: AI-Generated Data Can Poison Future AI Models

#66

Post by __loam »

coffeebeqn wrote: post #59
MacsHeadroom wrote: post #37

Earlier quoted context omitted.

This is exactly right. Model collapse does not exist in practice. In fact, LLMs trained on newer web scrapes have increased capabilities thanks to the generated output in their training data. For example, "base" pretrained models trained on scrapes which include generated outputs can 0-shot instruction follow and score higher on reasoning benchmarks. Intentionally produced synthetic training data takes this a step fu…

Wouldn’t reinforcement learning just weigh any nonsense data very low and then spammy garbage doesn’t really affect the model in the end much ? If the model and human experts can’t tell the difference then it’s probably pretty good AI generated data

Truth and what humans think is true are different things. Synthetic data was created by models that were trained to be convincing.

Re: AI-Generated Data Can Poison Future AI Models

#67

Post by __loam »

ToucanLoucan wrote: post #49
coldcode wrote: post #38

I think AI-generated images are worse for training AI generative models than LLMs, since there are so many now on the internet (see Instagram art related hashtags if you want to see nothing but AI art) compared to the quantity of images downloaded prior to 2021 (for those AI that did that). Text will always be more varied than seeing 10m versions of the same ideas that people make for fun. AI text can also be partial…

That's far from unique to instagram. I loathe Stable Diffiusion and co solely because they've utterly FLOODED every cool art-adjacent website with endless mediocre derivative shit. Like there was always low-effort content of course, but holy fuck, there is SO MUCH MORE now. And some of these people are trying to CHARGE for this uninspired junk!!!

I definitely think the flooding of art spaces is hugely problematic, but it is pretty funny to watch people try to "be an artist" by putting essentially no effort in. It definitely points to a lack of understanding in the field when all these people are basically generating a ton of images that are all derived from the same models. There's a lack of understanding of supply and demand, when the expectation is that your ai illustration that you made in like an hour with the same software as every other ai artist is that it's somehow going to be competitive on engagement with an original piece from an artist who has an audience. There's a lot of demand for artists like Mika Pikazo and Frank Frazetta, not the 100,000 ai artists out there.

Re: AI-Generated Data Can Poison Future AI Models

#69

Post by x86x87 »

nestorD wrote: post #41

I believe that this is a non-problem pushed forward by small-scale experiments that are not representative of what people actually do with AI generation. A lot of new content, while AI generated, has been hand picked and polished by a human (for example, while you might commit AI generated code to your codebase, you ensure that it is correct and follows your preferred style). Content farms will push gibberish out, bu…

i think it's pretty much a problem and it's going to ruin any chance of high quality, original content.

look at the original internet content and what seo has done to it. google and search in general results are trash nowadays. this is what genAI is going to do over long term. garbage in garbage out.

Re: AI-Generated Data Can Poison Future AI Models

#70

Post by x86x87 »

randcraw wrote: post #42

It's fascinating that error can accumulate through repeated trainings that 1) is undetected by humans and 2) can degrade LLM or diffusion models (or any transformer model?) so completely. This implies that not only do we not understand how latent knowledge is actually representated in deep nets, we don't know it forms or how it changes during training. If we did, we could have predicted the destructive impact of recy…

The effect is not new. We have known about it ever since we've had basic machine learning. The way to look at it is somewhat novel but not surprising at all.
Post reply on HN