Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security Threat Model Review of the Apple Child Safety Features [pdf]

apple.com

Post reply on HN
Print view View original

81–90 of 393 posts

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#81

Post by notJim »

almostdigital wrote: post #19

> Apple will publish a Knowledge Base article containing a root hash of the encrypted CSAM hash database included with each version of every Apple operating system that supports the feature. Additionally, users will be able to inspect the root hash of the en- crypted database present on their device, and compare it to the expected root hash in the Knowledge Base article. This is just security theater, they already si…

> Until a 1-line code change happens that hooks it into UIImage.

I really don't understand this view. You are using proprietary software, you are always an N-line change away from someone doing something you don't like. This situation doesn't change this.

If you only use open source software and advocate for others to do the same, I would understand it more.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#82

Post by drodgers »

> The second protection [against mis-inclusion of non CSAM hashes] is human review: there is no automated reporting in Apple’s system. All positive matches must be visually confirmed by Apple as containing CSAM before Apple will disable the account and file a report with the child safety organization.

I don't understand this at all. As I understand it, part of the problem is that — in the US — Apple isn't legally allowed to review, transmit or do anything else with suspected CSAM images, so they can't have a manual review process (or even check that their neural hashing is working as expected on the real dataset).

Does anyone else have any idea of what this is trying to describe?

If Apple really are somehow reviewing flagged photos to confirm that they're CSAM and not maliciously flagged files before sending any reports, then that does make the system substantially more resilient to Five Eyes abuse (not that I wish that job on anyone).

Edit: There's more context later in the document

> First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.

This is more confusing: visually comparison using a second perceptual hash doesn't actually to provide any protection against mis-inclusion of non-CSAM images: it just double-checks that the image really was a match in the database (ie. protects against hash-collision errors), but it does't check that the database contained actual CSAM.

Apple explicitly says that this process protects against mis-inclusion though, which doesn't make sense to me yet.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#83

Post by g42gregory »

kemayo wrote: post #22
almostdigital wrote: post #19

> Apple will publish a Knowledge Base article containing a root hash of the encrypted CSAM hash database included with each version of every Apple operating system that supports the feature. Additionally, users will be able to inspect the root hash of the en- crypted database present on their device, and compare it to the expected root hash in the Knowledge Base article. This is just security theater, they already si…

> And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, etc. Although this is true, the same argument already applies to "your phone might be scanning all your photos and stealthily uploading them" -- Apple having announced this program doesn't seem to have changed the odds of that. At some point you have to trust yo…

> At some point you have to trust your OS vendor.

Yes, and we were trusting Apple. And now this trust is going away.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#84

Post by jolux »

LexGray wrote: post #80
jolux wrote: post #38

Earlier quoted context omitted.

> And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, etc. They describe a process for third parties to audit that the database was produced correctly.

Do we have any idea how the NCMEC database is curated? Are there cartoons from Hustler depicting underage girls in distress? Green text stories stating they are true about illegal sexual acts? CGI images of pre-pubescent looking mythical creatures? Manga/Anime images which are sold on the Apple Store? Legitimate artistic images from books currently sold? Images of Winnie the Pooh the government has declared pornograp…

Apple is manually reviewing every case to ensure it’s CSAM. You do have to trust them on that.

But if your problem is with NCMEC, you’ve got a problem with Facebook and Google who are already doing this too. And you can’t go to jail for possessing adult pornography. So even if you assume adult porn images are in the database, and Apple’s reviewers decide to forward them to NCMEC, you would still not be able to be prosecuted, at least in the US. Ditto for pictures of Winnie the Pooh. But for the rest of what you describe, simulated child pornography is already legally dicey as far as I know, so you can’t really blame Apple or NCMEC for that.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#85

Post by FabHK »

cmsj wrote: post #57

This seems like a pretty helpful document, and really shows how carefully the CSAM system has been designed. Two things that stuck out to me: 1) The CSAM hash database is encrypted on-device 2) The device doesn't know the results of hash comparisons These suggest to me that the hash comparison is going to happen inside the Secure Enclave - give it the encrypted database and the NeuralHash of an image, it gives you ba…

> These suggest to me that the hash comparison is going to happen inside the Secure Enclave - give it the encrypted database and the NeuralHash of an image, it gives you back a Security Voucher to upload.

My understanding is that the hash comparison, or at any rate the counting of matches, is done in the cloud then. The way the crypto is set up is that the server has no access to the security envelope and its contents (and, after E2EE in the future, to any images) unless the number of matches exceeds the threshold.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#86

Post by robryk »

kevinsundar wrote: post #15
tehnub wrote: post #9

Their use of the phrase "This claim is subject to code inspection by security researchers like all other iOS device-side security claims" stood out to me. Could someone tell me how that inspection works? Are there researchers who are given the source code? (I posted this on another thread [0] earlier, but it's more relevant here) [0]: https://news.ycombinator.com/item?id=28175619

Apple actually gives special devices to Security Researchers that allow them further access into the device than a normal consumer device: https://developer.apple.com/programs/security-research-devic... In this way, third party security researchers can verify their claims. It actually works out pretty well for them since third party security researchers often find pretty severe vulnerabilities through this program.

That program, at least when it was introduced, required participants not to report security vulnerabilities publicly until Apple allowed them to do so, with no limits on how long that can be (see https://news.ycombinator.com/item?id=23920454 for a discussion from that time).

That makes this program particularly useless for the purpose of auditing whether Apple is adhering to its promises.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#87

Post by shuckles »

drodgers wrote: post #82

> The second protection [against mis-inclusion of non CSAM hashes] is human review: there is no automated reporting in Apple’s system. All positive matches must be visually confirmed by Apple as containing CSAM before Apple will disable the account and file a report with the child safety organization. I don't understand this at all. As I understand it, part of the problem is that — in the US — Apple isn't legally all…

Your understanding is incorrect. Apple can, and is in fact required to, verify that they have actual CSAM before forwarding it to the Cyber Tip line. At that point, they must delete the information within 60 days.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#88

Post by shuckles »

himaraya wrote: post #73
shuckles wrote: post #71

Earlier quoted context omitted.

And how is telling you in great detail about what they’re planning to do months before they do it and giving you a way to opt out in advance a breach of trust? What more did you expect from them?

You forgot, 'after it leaked'

It’s almost certain the “leak” was from someone they had pre-briefed prior to a launch. You don’t put together 80+ pages of technical documentation with multiple expert testimony in 16 hours.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#89

Post by shuckles »

himaraya wrote: post #65
shuckles wrote: post #58

Earlier quoted context omitted.

What’s the difference between hybrid cloud/local scanning “due to a bug” checking all your files and uploading too many safety vouchers and cloud scanning “due to a bug” uploading all your files and checking them there?

...because cloud uploads require explicit user consent, practically speaking? Apple's system requires none.

I think the mods should consider shadow banning you. Your comments make little sense.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#90

Post by LexGray »

lifty wrote: post #41

Is there a good argument for doing the scanning on the phone and not on the iCloud servers?

Photos are currently encrypted e2e already so Apple does not have the access server side to create the hash unlike their competition who chews their customers data for machine learning. https://support.apple.com/en-us/HT202303
Post reply on HN