Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tainting the CSAM client-side scanning database

blog.xot.nl

Post reply on HN
Print view View original

221–230 of 276 posts

Re: Tainting the CSAM client-side scanning database

#221

Post by Retr0id »

astrange wrote: post #199
Retr0id wrote: post #145

The article considers "an entity that is allowed to propose new entries to the CSAM database". You don't even need this! You could target a whole "social cluster" of people without having any special privileges within this system. As an example, lets say you want to attack environmental protesters. For image A, you create a meme about climate change. For image B, you procure something that looks, to humans, like CSAM…

> and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the database. I don't think NCMEC adds random images they find to the A1 list without knowing their origin. > until the average meme-savvy environmental protester's device gets flagged for further scrutiny. Who is this an attack on? A moderation contractor maybe, but no…

Regarding the first part, if that really is the relevant policy, fair enough. I don't know the specific policies used to include or exclude a given image. Where can I find the details?

I consider someone looking at pictures from my local device without my consent to be an attack on my privacy, regardless of the content, or whether they send me to jail afterwards.

The reason apple's threshold exists in the first place is because individual false-positives happen. Some of the images leaked from the victim's device may be entirely unrelated. In the specific example of an environmental protester, there might even be images documenting "crimes" (entirely unrelated to CSAM), due to increased criminalization of protest techniques.

A system that may be manipulated (anonymously, from great distance) to trigger spot-checks on the devices of anyone I don't like is a broken system.

Re: Tainting the CSAM client-side scanning database

#222

Post by LorenPechtel »

regularjack wrote: post #65
Hizonner wrote: post #43

It says: > This shows that the database can be tainted with non-CSAM material by an entity that can submit entries to it. Actually, it can easily be tainted by anybody . Take your massaged hash-colliding image, which remember is still visually child porn , and post it on some pedos-R-us forum. The people who maintain the database actively troll those forums. They'll see the image and add the hash to the database for…

You'd be publishing child porn, which I think is not the wisest thing to be doing.

This is probably a state-level actor doing it. They're not at risk.

Re: Tainting the CSAM client-side scanning database

#223

Post by bastawhiz »

pornel wrote: post #48
bastawhiz wrote: post #5

The issue described here, to my understanding, is that you find or create csam and then manipulate it so that its fingerprint collides with another image that you want to be flagged as csam. You then submit the manipulated version of the found or generated image to the authority. First, at what point does the authority go "uh... Where did you get this from?" Practically speaking, the people doing this would have to b…

Abusers who have actual CSAM could intentionally publish the fingerprints to sabotage the scanning scheme. If the illegal fingerprints become known, it will be possible to generate false positives and overwhelm verification/enforcement with bogus matches.

It's not an "any of us problem" for a bad system to be overwhelmed. Let it be overwhelmed.

Re: Tainting the CSAM client-side scanning database

#224

Post by LorenPechtel »

boerseth wrote: post #56
npteljes wrote: post #30

Earlier quoted context omitted.

Another version of the attack is to manipulate an innocent image in a way that's flagged by detection systems, and then spread that image somehow. If any of the responses are automatic, then it's a lot of false positives in the system, a lot of contact from police, disabling of the host, things like this.

There's also not really any need to trick the system or manipulate other files to generate collisions. Spreading one of the original law-breaking files to unwilling targets would work as a trigger just the same.

But the recipient would know they had gotten CSAM and wouldn't pass it along.

Re: Tainting the CSAM client-side scanning database

#225

Post by bastawhiz »

baby_souffle wrote: post #23
bastawhiz wrote: post #5

The issue described here, to my understanding, is that you find or create csam and then manipulate it so that its fingerprint collides with another image that you want to be flagged as csam. You then submit the manipulated version of the found or generated image to the authority. First, at what point does the authority go "uh... Where did you get this from?" Practically speaking, the people doing this would have to b…

Would this org refuse to accept CSAM from an anonymous source just because they can’t point to its origin?

I think they'd send some police to find out

Re: Tainting the CSAM client-side scanning database

#226

Post by LorenPechtel »

londons_explore wrote: post #60
chatmasta wrote: post #32

Earlier quoted context omitted.

I assume the answer to that will be that there is no need to differentiate between them. And honestly, I agree with that argument. Possession of CSAM should be illegal regardless of whether it's "real" or not. But the proposed scanning system is the wrong solution, regardless of any "real or AI" ambiguity, because it's possible to generate false positives with nonsense images that aren't even close to the expected CS…

> I assume the answer to that will be that there is no need to differentiate between them. And honestly, I agree with that I disagree. The point is to reduce actual child abuse. The images are in a way only tangential. If an image is made with an AI with no actual child being abused, then it shouldn't be a crime. In a way, it's better , because it will distract the crowd of people into this sort of stuff from activit…

That's how I see it, also. There is a very clear pattern that prevalence of pornography reduces rape. Why in the world should we expect a different result when we narrow the context?

Yucky as it is I believe the answer here is to have image-generating AIs sign their work. Something properly signed is known not to involve any actual children and would thus be legal.

(My philosophy in general is that for something to be illegal the state should be able to show a non-consenting victim or the undue risk of a victim (ie, DUI). I do not believe disgusting things in private warrant a law.)

Re: Tainting the CSAM client-side scanning database

#227

Post by LorenPechtel »

anigbrowl wrote: post #141
londons_explore wrote: post #60

Earlier quoted context omitted.

> I assume the answer to that will be that there is no need to differentiate between them. And honestly, I agree with that I disagree. The point is to reduce actual child abuse. The images are in a way only tangential. If an image is made with an AI with no actual child being abused, then it shouldn't be a crime. In a way, it's better , because it will distract the crowd of people into this sort of stuff from activit…

You're assuming the truth of your conclusion without testing it. It's equally possible that encountering AI-CSAM is going to incentivize collectors to pay a premium for 'the real stuff', just as many CSAM collectors end up getting caught when they try to make the leap into engaging in abusive activities for real. Your mental model of how CSAM enthusiasts think isn't anchored in reality.

I don't think that's how it works. It's not that CSAM drives them to actual abuse, but that for some CSAM isn't enough to sufficiently satisfy their desires that they go on to real abuse.

Thus I see no reason they would differentiate. With normal adult porn do we care that makeup and such might be involved?

Re: Tainting the CSAM client-side scanning database

#228

Post by LorenPechtel »

Chabsff wrote: post #63
londons_explore wrote: post #60

Earlier quoted context omitted.

> I assume the answer to that will be that there is no need to differentiate between them. And honestly, I agree with that I disagree. The point is to reduce actual child abuse. The images are in a way only tangential. If an image is made with an AI with no actual child being abused, then it shouldn't be a crime. In a way, it's better , because it will distract the crowd of people into this sort of stuff from activit…

You may think that intuitively, but actual studies actually indicate the opposite. Usage of CSAM material leads to increased risks of contacting and abusing children. This needs to be balanced against rights to privacy and expression, which I personally think take precedence, but pretending that it can serve as harm reduction is just not correct.

Of course it's related. That doesn't mean it "leads to"--I think that's a case of the cart before the horse.

Those who have no sexual interest in children are neither going to have CSAM nor engage in abuse. The fact that they had CSAM already shows it's a highly non-random sample. The control would be pedophiles with no access to CSAM--but how do you find that control group????

Re: Tainting the CSAM client-side scanning database

#229

Post by LorenPechtel »

chatmasta wrote: post #71
Chabsff wrote: post #63

Earlier quoted context omitted.

You may think that intuitively, but actual studies actually indicate the opposite. Usage of CSAM material leads to increased risks of contacting and abusing children. This needs to be balanced against rights to privacy and expression, which I personally think take precedence, but pretending that it can serve as harm reduction is just not correct.

I'm personally highly skeptical of the "offering them an outlet" argument. I'd be less suspicious of the idea if its proponents also suggested limiting it to controlled settings, e.g. during meetings with a professional psychiatrist. But I'm sorry, I just don't believe anyone holed up in their room with a bunch of fake CSAM is "just using it as an outlet" or "protecting real kids from harm." I mean, it almost sounds…

Pornography reduces rape.

Violent movies that appeal to teens reduce vandalism and the like--they're in the theater rather than out causing trouble. (And it's not displaced, rates don't spike later, they just return to normal.)

Re: Tainting the CSAM client-side scanning database

#230

Post by kelnos »

Hizonner wrote: post #176
GuB-42 wrote: post #169

Earlier quoted context omitted.

That would be ridiculous. But there are penalties for false claims. There is a fine for claiming copyright you don't own, and if you go further and ask for takedowns, you are also liable for damage. The problem is that these are rarely enforced. Even a $100 fine for a false claim on YouTube would be enough to weed out bots and click farms. And for the most serious cases, have the infringer pay damage and a bigger fin…

> That would be ridiculous. Why is it ridiculous, in a world where the penalty for sharing a single music file is $250,000?

That's an argument to reduce the penalty for sharing a single music file, not for making the penalty for a false report also ridiculous.

Then again, I personally do think that knowingly filing a false report of law-breaking should be treated as a very serious crime. I think the harm of filing a false report of copyright infringement is greater than the act of infringing copyright itself.

Post reply on HN