Apple clarifies why it abandoned plan to detect CSAM in iCloud photos • Hacker News Board

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#251

Post by seanieb » Sat, Sep 02, 2023, 6:54 AM UTC

akira2501 wrote: post #156

stuartjohnson12 wrote: post #57

Earlier quoted context omitted.

You'd be surprised. Lots of offenders are very low sophistication. If you read news articles about how a particular offender was caught with illegal material, so so often it's because they uploaded it to a cloud provider. It's not a one-sided tradeoff here.

What percentage of offenders victimize children and never record it in any way? If that's the overwhelming majority of abuse cases, what are we even doing here?

Worse. Only a tiny fraction of child abuse material cases actually get investigated due to lack of resources… this debate about scanning is an insane distraction.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#252

Post by theshrike79 » Sat, Sep 02, 2023, 6:55 AM UTC

barsonme wrote: post #240

judge2020 wrote: post #213

Earlier quoted context omitted.

> What do you think they were going to do once the scanning turned up a hit? Access the photos? Well that negates the first statement. In the whitepaper, the cryptography required that Apple have multiple different photodna (or whatever the name was for the on-device one) matches before they could unwrap the user's message containing these suspected CSAM photos and to then send them to NCMEC.

Also, IIRC, it wasn’t the raw photos. It was small thumbnails of them.

"reduced-quality copy" was the wording in the whitepaper IIRC.

So the resolution most likely would've been the same, but the detail blurred so that the poor human agent wouldn't have to see actual CSAM, just enough to make a call whether it is or isn't a likely match.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#253

Post by theshrike79 » Sat, Sep 02, 2023, 6:59 AM UTC

Moldoteck wrote: post #143

theshrike79 wrote: post #133

Earlier quoted context omitted.

You do know that we currently have "thousands of people" watching for and tagging the most heinous shit people upload to social media, right? There are multiple sources for this how we use outsourced people from Africa and Asia to weed through all of the filth people upload on FB alone. "Looking illegal" isn't enough to trigger a CSAM check in this case. It's perfectly normal to take pictures of your own kids without…

Again, somebody can train an algorithm to create false positives or real csam like pictures, like close enough to trigger the check. Afaik csam is not about exact match but rather close enough match based on a clever hashing algorithm, and in this case, algorithm can be induced into false positives(and by my limitet knowledge, hashing can have collisions) or even true positives but that are fully generated(and afaik…

People DID get into a huge fuss and started building false positive generators in a huge wave. Like they were proving something or pwning apple.

Nobody read the bit about an actual human verifying results before any law enforcement would be called in.

And outsourcing checking is a huge industry even today[0]. How do you think the huge social media companies keep CSAM, gore etc out of their systems? They're not using Pied Piper's hotdog or not algorithm, that's for sure.

[0] https://www.theverge.com/2019/2/25/18229714/cognizant-facebo...

A snippet from the article:

> The video depicts a man being murdered. Someone is stabbing him, dozens of times, while he screams and begs for his life. Chloe’s job is to tell the room whether this post should be removed.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#254

Post by simondotau » Sat, Sep 02, 2023, 7:22 AM UTC

guiambros wrote: post #238

simondotau wrote: post #231

Earlier quoted context omitted.

Apple was concerned about governments using the excuse of CSAM to pass laws which would force Apple to weaken encryption across the board. Whether this was the right response to such concern is something I’m not unsympathetic towards. Certainly I think it’s reasonable to say that Apple was trying to thread a needle in a way which was never going to please everyone, even if it somehow turns out to have been the least-…

Yes, but to OP's point: this was patently obvious from the onset. Even here the comments at the time [1] pointed to all sorts of potential misuse, political or religious prosecution, dystopian cases of false positives, and that this would leave the door open to future government escalation beyond CSAM. How could they not see that they would have a giant backlash on their hands? Did they overestimate their ability to…

I think Apple was doing the exact opposite. They wanted to do the __least possible thing__ in order to stave away the far worse outcome of intelligence departments using the "it's for our children" excuse to pressure elected representatives to vote for back doors on consumer encryption.

The ridiculous thing is that Apple's proposal was functionally identical to what other platform vendors (e.g. Google, Microsoft) were already doing. In all cases — including Apple's proposed system — only photos uploaded to cloud storage would be scanned to see if it matched CSAM already known to the government. The only difference with Apple's proposal was initial "fuzzy hash" calculation would be performed on-device prior to upload, instead of on-cloud after upload.

The reason for doing it differently was because it meant (in theory) satisfying both masters — implementing real end-to-end encryption, while not being seen as a CSAM scanning laggard compared to Google, Microsoft, etc.

Other vendors just scan all your shit and nobody cares.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#255

Post by olliej » Sat, Sep 02, 2023, 7:23 AM UTC

robertoandred wrote: post #236

olliej wrote: post #4

Earlier quoted context omitted.

No, the terrible misfeature that this group wants is “government provides a bunch of opaque hashes that are ‘CSAM’, all images are compared with those hashes, and if the hashes match then the user details are given to police” Note that by design the hashes cannot be audited (though in the legitimate case I don’t imagine doing so would be pleasant), so there’s nothing stopping a malicious party inserting hashes of any…

How exactly would you be able to "filter" LGBT content? I don't think you understand how this system would've worked.

Hypothetically you have hashes for two people of gender X (lets be honest, based on popularity of different types of porn two men). This is not meaningfully different from opaque hash of "CSAM".

But you're missing the point:

Step 1. generate some opaque hash of the "semantics" of an image

Step 2. compare those hashes to some list of hashes of "CSAM", which again fundamentally cannot be audited

Step 3. report any hits to law enforcement

Step 4. person X is being investigated due to reported violations of laws against child abuse.

Basically: how do you design a system in which the state provides "semantic" hashes of "CSAM" that cannot be trivially abused by inclusion of non-CSAM as "CSAM", or by laws mandating inclusion of things that are objectively not-CSAM. Hypothetically: hashes that match christian crosses, star of David, muslim star and/or crescent, etc. Or in the US DNC, RNC, pride, etc flags. Recall that definitionally no one can audit the hashes that would trigger notifying law encforcement.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#256

Post by no_time » Sat, Sep 02, 2023, 7:43 AM UTC

judge2020 wrote: post #211

no_time wrote: post #52

Earlier quoted context omitted.

> I’m not sure I understand apples logic here. Are iCloud Photos in their data centers not scanned? Isn’t everything by default for iCloud users sent there automatically to begin with? Doesn’t the same logic around slippery slope also apply to cloud scans? I don’t see the problem with this status quo. There is a clear demarcation between my device and their server. Each serving the interests of their owner. If I have…

This status quo is that a lot of countries want to use the CSAM argument to push privacy-invasive technology (cough UK) like e.g. forcing companies to allow the government to break E2EE to catch CSAM distributors. Apple made this feature while planning to move iCloud Photos to E2EE so that they could argue "look, we still catch x CSAM distributors with n < 0.x% false positive rate, even with E2EE photos. therefore yo…

I know "give them an inch, they take a mile" is a reductive comparison but I really can't see this way of thinking going any other way in the long term.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#257

Post by turquoisevar » Sat, Sep 02, 2023, 8:25 AM UTC

salawat wrote: post #230

turquoisevar wrote: post #221

Earlier quoted context omitted.

I was looking for that! Great addition to provide more context.

Don't forget this part: >(e) Failure To Report.—A provider that knowingly and willfully fails to make a report required under subsection (a)(1) shall be fined— (1) in the case of an initial knowing and willful failure to make a report, not more than $150,000; and (2) in the case of any second or subsequent knowing and willful failure to make a report, not more than $300,000. I find these clauses at odds with one anot…

It’s because they wanted their cake and eat it too, get as close as possible to the 4th amendment without crossing the line.

Put simply, if they have knowledge of it they have a duty to report, but they can’t be compelled to try and find out.

In theory this means that if they happen to stumble upon it or are being alerted to it by a third party (e.g. user report) then they have to report it, in practice many voluntarily monitor it, maybe because they want to avoid having to litigate that they didn’t have knowledge of it or maybe because it’s good PR or maybe because they care for the case.

I think in most cases it’s all of the above in one degree or another.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#258

Post by smittywerben » Sat, Sep 02, 2023, 8:26 AM UTC

It surprises me that people think their common $CLOUD_STORAGE_PROVIDER doesn't already scan files for child exploitation material.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#259

Post by withinboredom » Sat, Sep 02, 2023, 8:57 AM UTC

tzs wrote: post #193

I'm curious about the new parental control features they announced at the same time as the iCloud photo scanning. My recollection is that when they withdrew the iCloud scanning they also withdrew the new parental controls. I'm curious why they also withdrew those. For those who don't remember the parental control, which were largely overshadowed by the controversy over the cloud stuff, they were to work like this: 1.…

This feature is back in iOS 17, but it doesn’t do notifications and can be enabled per-device. It basically just blurs nsfw photos when they’re on the screen. It’s great when on a business trip and sexting with the SO. You can open your messages in front of your coworkers without having to worry about the messages being open from the night before.

Re: Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

#260

Post by nottorp » Sat, Sep 02, 2023, 9:04 AM UTC

advisedwang wrote: post #190

beej71 wrote: post #135

I think they likely also considered the lawsuit exposure. If just 0.0001% of users sued over false positives, Apple would be in serious trouble. And there's another dynamic where telling your customers you're going to scan their content for child porn is the same as saying you suspect your customers of having child porn. And your average non-criminal customer's reaction to that is not positive for multiple reasons.

Section 230 removes liability for restricting good faith attempts to combat CSAM. > (2) Civil liability > No provider or user of an interactive computer service shall be held liable on account of— > (A) any action voluntarily taken in good faith to restrict access to or availability of material that the provider or user considers to be obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise ob…

I don't see any reference to child porn there. Who decides what's obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable? Especially otherwise objectionable?