Security Threat Model Review of the Apple Child Safety Features [pdf] • Hacker News Board

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#151

Post by XorNot » Sat, Aug 14, 2021, 1:51 AM UTC

slg wrote: post #51

kemayo wrote: post #22

Earlier quoted context omitted.

> And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, etc. Although this is true, the same argument already applies to "your phone might be scanning all your photos and stealthily uploading them" -- Apple having announced this program doesn't seem to have changed the odds of that. At some point you have to trust yo…

Which is why I am confused by a lot of this backlash. Apple already controls the hardware, software, and services. I don't see why it really matters where in that chain the scanning is done when they control the entire system. If Apple can't be trusted with this control today, why did people trust them with this control a week ago?

Because people (HN especially) overestimate how easy it is to develop and deploy "stealth" software that is never detected in a broad way.

The best covert exfiltration is when you can hit individual devices in a crowd, so people already have no reason to be suspicious. But you're still leaving tracks - connections, packet sizes etc. if you actually want to do anything and you only need to get caught once for the game to be up.

This on the other hand is essentially the perfect channel for its type of surveillance...because it is a covert surveillance channel! Everyone is being told to expect it to exist, that it's normal, and that it will receive frequent updates. No longer is their a danger a security researcher will discover it, its "meant" to be there.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#152

Post by shuckles » Sat, Aug 14, 2021, 1:54 AM UTC

stale2002 wrote: post #150

shuckles wrote: post #71

Earlier quoted context omitted.

And how is telling you in great detail about what they’re planning to do months before they do it and giving you a way to opt out in advance a breach of trust? What more did you expect from them?

> What more did you expect from them? Well they could not do it.

You might prefer that, but it doesn’t violate your privacy for them to prefer a different strategy.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#153

Post by 1f60c » Sat, Aug 14, 2021, 1:55 AM UTC

Having read the document, I am even more confident in Apple's approach than before. I commend Apple for its transparency.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#154

Post by chipotle_coyote » Sat, Aug 14, 2021, 1:58 AM UTC

new299 wrote: post #134

chipotle_coyote wrote: post #92

In other HN comments on this subject I've (hopefully) made it clear that I'm not really in favor of this project of Apple's, and that there's a legitimate "slippery slope" argument to be made here. So I hope people will entertain a contrarian question without downvoting me into oblivion. :) Here's the thing I keep circling around: assume that bad actors, government or otherwise, want to target political dissidents us…

> there's a legitimate "slippery slope" argument The slippery slope argument is the only useful argument here. The fundamental issue with their PSI/CSAM system is that they already were scanning iCloud content [1] and that they're seemingly not removing the ability to do that. If the PSI/CSAM system had been announced along side E2E encryption for iCloud backups, it would be clear that they were attempting to act in…

> If the PSI/CSAM system had been announced along side E2E encryption for iCloud backups, it would be clear that they were attempting to act in their users best interests.

Absolutely. I don't know whether there's a reason for this timing (that is, if they are planning E2E encryption, why they announced this first), but this is probably the biggest PR bungle Apple has had since "you're holding it wrong," if not ever.

> Apple are clearly willing to add functionality at the request of governments, on device.

Maybe? I'm not as willing to state that quite as definitively, given the pushback Apple gave in the San Bernardino shooter case. Some of what their Privacy Engineering head said in that TechCrunch article suggests that Apple has engineered this to be strategically awkward, e.g., generating the hashes by using ML trained on the CSAM data set (so the hashing system isn't as effective on other data sets) and making the on-device hashing component part of the operating system itself rather than a separately updatable data set. That in turn suggests to me Apple is still looking for an engineering way to say "no" if they're asked "hey, can you just add these other images to your data set." (Of course, my contention that this is not ultimately an engineering problem applies here, too: even if I'm right about Apple playing an engineering shell game here, I'm not convinced it's enough if a government is sufficiently insistent.)

A minor interesting tidbit: your linked Sophos story is based on a Telegraph UK story that has this disclaimer at the bottom:

> This story originally said Apple screens photos when they are uploaded to iCloud, Apple's cloud storage service. Ms Horvath and Apple's disclaimer did not mention iCloud, and the company has not specified how it screens material, saying this information could help criminals.

It's hard to say what they were actually doing, but it's reasonable to suspect it's an earlier, perhaps entirely cloud-based rather than partially cloud-based, version of NeuralHash.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#155

Post by bnj » Sat, Aug 14, 2021, 2:01 AM UTC

A curious outcome of the scrutiny that the system design is receiving is that it will make it less effective at the task of catching actual collections of CSAM.

Page eleven promises something I haven’t seen before: that Apple will publish the match threshold for each version. Meaning, even if after all this, there are abusers still using iCloud photos to manage their illegal collection of CSAM, they can check back to know what number of images is safe to keep.

A twisted version of the Streisand effect.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#156

Post by chipotle_coyote » Sat, Aug 14, 2021, 2:01 AM UTC

pseudalopex wrote: post #148

chipotle_coyote wrote: post #92

In other HN comments on this subject I've (hopefully) made it clear that I'm not really in favor of this project of Apple's, and that there's a legitimate "slippery slope" argument to be made here. So I hope people will entertain a contrarian question without downvoting me into oblivion. :) Here's the thing I keep circling around: assume that bad actors, government or otherwise, want to target political dissidents us…

Lots of people have made policy arguments. No US law requires client side scanning. No US law forbids E2E encryption. US courts don't let law enforcement agencies just demand everything they want from companies. Apple relied on that 5 years ago successfully.[1] And capitulating preemptively is bad strategy usually. What Neuenschwander said doesn't establish it isn't just an arbitrary limitation. [1] https://en.wikipe…

None of the laws do yet. My observation isn't about the laws as they necessarily exist now, just as the worry about how this could be abused isn't about Apple's policy as it exists now.

If we trust US courts to stop law enforcement agencies from demanding everything they want from companies, they they can stop law enforcement agencies from demanding Apple add non-CSAM data to the NeuralHash set. If we don't trust the courts to do that, then we're kind of back at square one, right?

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#157

Post by stale2002 » Sat, Aug 14, 2021, 2:02 AM UTC

shuckles wrote: post #152

stale2002 wrote: post #150

Earlier quoted context omitted.

> What more did you expect from them? Well they could not do it.

You might prefer that, but it doesn’t violate your privacy for them to prefer a different strategy.

why even ask the question " What more did you expect from them?" if you didn't care about the answer?

I gave a pretty obvious and clear answer to that, and apparently you didn't care about the question in the first place, and have now misdirected to something else.

I am also not sure what possible definition of "privacy" that you could be using, that would not include things such as on device photo scanning, for the purpose of reporting people to the police.

Like, lets say it wasn't Apple doing this. Lets say it was the government. As in, the government required every computer that you own, to be monitored for certain photos, at which point the info would be sent to them, and they would arrest you.

Without a warrant.

Surely, you'd agree that this violates people's privacy? The only difference in this case, is that the government now gets to side step 4th amendment protections, by having a company do it instead.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#158

Post by somebodythere » Sat, Aug 14, 2021, 2:04 AM UTC

jolux wrote: post #84

LexGray wrote: post #80

Earlier quoted context omitted.

Do we have any idea how the NCMEC database is curated? Are there cartoons from Hustler depicting underage girls in distress? Green text stories stating they are true about illegal sexual acts? CGI images of pre-pubescent looking mythical creatures? Manga/Anime images which are sold on the Apple Store? Legitimate artistic images from books currently sold? Images of Winnie the Pooh the government has declared pornograp…

Apple is manually reviewing every case to ensure it’s CSAM. You do have to trust them on that. But if your problem is with NCMEC, you’ve got a problem with Facebook and Google who are already doing this too. And you can’t go to jail for possessing adult pornography. So even if you assume adult porn images are in the database, and Apple’s reviewers decide to forward them to NCMEC, you would still not be able to be pro…

Drawings are prosecutable in many countries including Canada, the UK, and Australia. Also, iCloud sync is enabled by default when you set up your device, whereas the Facebook app at least is sandboxed and you have to choose to upload your photos.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#159

Post by Retric » Sat, Aug 14, 2021, 2:05 AM UTC

pseudalopex wrote: post #148

chipotle_coyote wrote: post #92

In other HN comments on this subject I've (hopefully) made it clear that I'm not really in favor of this project of Apple's, and that there's a legitimate "slippery slope" argument to be made here. So I hope people will entertain a contrarian question without downvoting me into oblivion. :) Here's the thing I keep circling around: assume that bad actors, government or otherwise, want to target political dissidents us…

Lots of people have made policy arguments. No US law requires client side scanning. No US law forbids E2E encryption. US courts don't let law enforcement agencies just demand everything they want from companies. Apple relied on that 5 years ago successfully.[1] And capitulating preemptively is bad strategy usually. What Neuenschwander said doesn't establish it isn't just an arbitrary limitation. [1] https://en.wikipe…

That’s different. The FBI can legally require Apple or any other US company to search for specific files it has access to on it’s own servers because nothing currently shields backup providers. They could and did force Apple to aid in unlocking iPhones when Apple had that capacity. What they couldn’t do was “These orders would compel Apple to write new software that would let the government bypass these devices' security and unlock the phones.”

Forcing companies to create back doors in their own is legally a very different situation. As to why iCloud is accessible by Apple, the point is to backup a phone someone lost. Forcing people to keep some sort of key fob with a secure private key safe in order to actually have access to their backups simply isn’t tenable.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#160

Post by cinquemb » Sat, Aug 14, 2021, 2:08 AM UTC

chipotle_coyote wrote: post #146

xvector wrote: post #132

Earlier quoted context omitted.

> It was illegal to export "strong encryption" for many years, remember? I've seen multiple reports that European lawmakers are planning to require some kind of scanning for CSAM. If this goes into effect, technology isn't going to block those laws for you. Your Purism phone will either be forced to comply or be illegal. The point is that with a Purism phone or custom ROM on my Android phone, I could disable these "l…

> The point is that with a Purism phone or custom ROM on my Android phone, I could disable these "legally required" features, because the law is fucking dumb, and my rights matter more. But if we assume a government is determined to do this, can't they find other ways to do it? If you were using Google Photos with your Purism phone, it doesn't matter what you do on your device. And you can say "well, I wouldn't use t…

> And then you can say, "But I'm using encryption," and the government mandates that they have a back door into all encrypted traffic that goes through their borders.

And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this.

I'm personally really interested in router level malware + 0days on devices as distribution vectors rather than the typical c&c setup.

> They'll go after people hosting it.

Not too hard to imagine one being able to distribute such things across millions of ephemeral devices that are networked and incentivized to host it, all across the world, regardless of illegality in any particular jurisdiction. Technology enables this, without such, it wont be possible.

> I think hackers and engineers really, really want to believe that math trumps policy, but it doesn't

I don't think that at all, I think it comes down to incentives. I was listening to a talk the other day where someone mentioned that for the longest time (since at least wwII), governments pretty much had a monopoly on cryptographers and now there are lots of places/systems that are willing to pay more to apply cutting edge research.

> but let's also fight for government policy that supports it, rather than assuming encryption and open source is a guarantee we can circumvent bad policy.

Much more cheaper for an individual, with more immediate feedback mechanisms doing one vs another. One can also scale a lot faster than another esp since one is very much divorced from implementation.