Security Threat Model Review of the Apple Child Safety Features [pdf] • Hacker News Board

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#321

Post by shapefrog » Sat, Aug 14, 2021, 9:38 AM UTC

xvector wrote: post #132

chipotle_coyote wrote: post #92

In other HN comments on this subject I've (hopefully) made it clear that I'm not really in favor of this project of Apple's, and that there's a legitimate "slippery slope" argument to be made here. So I hope people will entertain a contrarian question without downvoting me into oblivion. :) Here's the thing I keep circling around: assume that bad actors, government or otherwise, want to target political dissidents us…

> It was illegal to export "strong encryption" for many years, remember? I've seen multiple reports that European lawmakers are planning to require some kind of scanning for CSAM. If this goes into effect, technology isn't going to block those laws for you. Your Purism phone will either be forced to comply or be illegal. The point is that with a Purism phone or custom ROM on my Android phone, I could disable these "l…

> Apple, knowing that it is a private company completely and utterly incapable of resisting serious government demands (ie GCBD in China) should never have developed this capability to begin with.

You are basically arguing that an iphone be incablable of doing any function at all.

Dont want the government demanding the sent/recieved data - no data sending and recieving funcitons.

Dont want the government demanding phone call intercepts - no phone call functionality.

Dont want the government demanding the contents of the screen - no screen.

The pandoras box was opened the day someone inseted a radio chip and microphone into a device. It has been open for a very long time, this is not the moment it suddenly opened.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#322

Post by simondotau » Sat, Aug 14, 2021, 9:45 AM UTC

0xy wrote: post #320

simondotau wrote: post #311

Earlier quoted context omitted.

I clicked on your link hoping for serious analysis. I would have settled for interesting analysis. I was disappointed. It's just a guy who found one MD5 hash collision. The paragraph was written in a way that makes it unclear whether source of this specific hash was in fact NCMEC or if it was "other law enforcement sources". So which was it? Did this person follow up with the source to confirm whether his hash match…

It was not a hash collision. It was a false positive. The way the hashing solution works doesn't really allow for a false positive except in extraordinarily rare circumstances. It matched a man holding a monkey full clothed as a CSAM image, direct from NCMEC's database. He encountered a 20% false positive rate while running a moderately popular image service, with an admittedly low sample size. It's still evidence, a…

> It was not a hash collision. It was a false positive.

Again, this was an MD5. It's literally impossible to assert that it was a false positive with absolute certainty. A hash collision is not outside the realm of possibility, especially with MD5. Apparently no attempt was made to chase this up. And we still don't know whether it was the NCMEC database or "other law enforcement sources".

You continue to claim that it was "direct from NCMEC's database" but again, that isn't asserted by your source.

> He encountered a 20% false positive rate

He encountered one potential false positive. Converting one data point into a percentage is exactly why earlier I described this nonsense as being disingenuous. The fact that you would cite this source and then defend their statistical clown show is, in my opinion, strong positive evidence that your other citation-free assertions are all entirely made up.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#323

Post by dwaite » Sat, Aug 14, 2021, 9:52 AM UTC

laserlight wrote: post #312

dwaite wrote: post #297

Earlier quoted context omitted.

> I would say that it is a back door, because it would work even if iCloud Photos were E2E encrypted. Backdoor is defined by the Oxford Dictionary as "a feature or defect of a computer system that allows surreptitious unauthorized access to data." The system in question requires you to upload the data to iCloud Photos for the tickets to be meaningful and actionable. Both your phone and iCloud services have EULA which…

I see. My interpretation doesn't hold up given your definitions of back door. I bet the authorities would be happy with a surveillance mechanism disclosed in the EULA, though. Even if such a system is not technically a back door, I am opposed to it and would prefer Apple to oppose it. Edit: I just noticed that you had already clarified your argument in other replies. I am sorry to make you repeat it.

It has proven very difficult to oppose laws meant to deter child abuse and exploitation.

Note that while the EFF's mission statement is about defending civil liberties, they posted two detailed articles about Apple's system without talking about the questionable parts of the underlying CSAM laws. There was nothing about how the laws negatively impact civil liberties and what the EFF might champion there.

The problem is that the laws themselves are somewhat uniquely abusable and overreaching, but they are meant to help reduce a really grotesque problem - and reducing aspects like detection and reporting is not going to be as effective against the underlying societal issue.

Apple has basically been fighting this for the 10 years since the introduction of iCloud Photo, saying they didn't have a way to balance the needs to detect CSAM material without impacting the privacy of the rest of users. PhotoDNA was already deployed at Microsoft and being deployed by third parties like Facebook when iCloud Photo launched.

Now it appears that Apple was working a significant portion on that time toward trying to build a system that _did_ attempt to accomplish a balance between social/regulatory responsibility and privacy.

But such a system has to prop technical systems and legal policies against one another to make up the shortcomings of each, which make it a very complex and nuanced system.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#324

Post by 0xy » Sat, Aug 14, 2021, 10:22 AM UTC

simondotau wrote: post #322

0xy wrote: post #320

Earlier quoted context omitted.

It was not a hash collision. It was a false positive. The way the hashing solution works doesn't really allow for a false positive except in extraordinarily rare circumstances. It matched a man holding a monkey full clothed as a CSAM image, direct from NCMEC's database. He encountered a 20% false positive rate while running a moderately popular image service, with an admittedly low sample size. It's still evidence, a…

> It was not a hash collision. It was a false positive. Again, this was an MD5. It's literally impossible to assert that it was a false positive with absolute certainty. A hash collision is not outside the realm of possibility, especially with MD5. Apparently no attempt was made to chase this up. And we still don't know whether it was the NCMEC database or "other law enforcement sources". You continue to claim that i…

What you're missing is the statistical probability of two MD5 hashes colliding, which is astronomically unlikely.

Your argument is essentially that a collision is more likely than a false positive, which would imply a false positive rate of 0.00% in NCMEC's database based on its size.

It's clear that nothing will convince you if you believe that humans managing a database will never, ever make a mistake after over 300,000,000 entries. Because if they make one single mistake, then it supports my argument -- a false positive becomes substantially more likely statistically than a hash collision.

You're also providing a pretty large red herring with your suggestion that he could've simply asked NCMEC if it was a false positive or a hash collision. NCMEC would never provide that information, because the database is highly secret.

Given those statistics, I think that source is more than valid.

>in my opinion, strong positive evidence that your other citation-free assertions are all entirely made up

I am happy to let you believe that I'm lying.

One industry insider whose employer works with NCMEC cited a false positive rate of 1 in 1,000. [1] The product he works in is used in conjunction with NCMEC's database. Elsewhere, in press releases, the company cites a failure rate of 1% (presumably both false positives and false negatives) [2]

[1] https://news.ycombinator.com/item?id=21446562

[2] https://www.prnewswire.com/news-releases/thorns-automated-to...

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#325

Post by arthur_sav » Sat, Aug 14, 2021, 10:45 AM UTC

chipotle_coyote wrote: post #92

In other HN comments on this subject I've (hopefully) made it clear that I'm not really in favor of this project of Apple's, and that there's a legitimate "slippery slope" argument to be made here. So I hope people will entertain a contrarian question without downvoting me into oblivion. :) Here's the thing I keep circling around: assume that bad actors, government or otherwise, want to target political dissidents us…

Absolutely.

We've seen this time after time where the 3 letter agencies give companies an ultimatum: either comply or get shut down.

The worse part is that nobody can do anything about it. Under the cloak of secrecy and threats a faceless government is shaping major policy, breaking laws etc...

My only hope is that the biggest companies can speak up since they have a bit of a leverage. They can rally people behind them if the requests are borderline unethical, like spying on everybody.

If Apple can't deal with this, NOBODY else can.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#326

Post by simondotau » Sat, Aug 14, 2021, 10:56 AM UTC

0xy wrote: post #324

simondotau wrote: post #322

Earlier quoted context omitted.

> It was not a hash collision. It was a false positive. Again, this was an MD5. It's literally impossible to assert that it was a false positive with absolute certainty. A hash collision is not outside the realm of possibility, especially with MD5. Apparently no attempt was made to chase this up. And we still don't know whether it was the NCMEC database or "other law enforcement sources". You continue to claim that i…

What you're missing is the statistical probability of two MD5 hashes colliding, which is astronomically unlikely. Your argument is essentially that a collision is more likely than a false positive, which would imply a false positive rate of 0.00% in NCMEC's database based on its size. It's clear that nothing will convince you if you believe that humans managing a database will never, ever make a mistake after over 30…

Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data. I am merely pushing back at your evidence-free claim that "it's full of false positives."

Once again, you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources". You're reading far more into your sources than they are saying.

And now you are conflating claims of false positives in the origin database with rates of false positives in a perceptual hashing algorithm. You are clearly very confused.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#327

Post by ClumsyPilot » Sat, Aug 14, 2021, 11:01 AM UTC

arthur_sav wrote: post #325

chipotle_coyote wrote: post #92

In other HN comments on this subject I've (hopefully) made it clear that I'm not really in favor of this project of Apple's, and that there's a legitimate "slippery slope" argument to be made here. So I hope people will entertain a contrarian question without downvoting me into oblivion. :) Here's the thing I keep circling around: assume that bad actors, government or otherwise, want to target political dissidents us…

Absolutely. We've seen this time after time where the 3 letter agencies give companies an ultimatum: either comply or get shut down. The worse part is that nobody can do anything about it. Under the cloak of secrecy and threats a faceless government is shaping major policy, breaking laws etc... My only hope is that the biggest companies can speak up since they have a bit of a leverage. They can rally people behind th…

"either comply or get shut down'

I think if a three letter agency 'shuts down' Apple, the political blowback will be nuclear. The agency will get put on a leash if they pull some shit like tthat

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#328

Post by tzs » Sat, Aug 14, 2021, 11:03 AM UTC

makecheck wrote: post #139

I don’t like the idea of stuff running on my device, consuming my battery and data, when the only point is to see if I am doing something wrong? An analogy I can come up with is: the government hires people to visit your house every day, and while they’re there they need your resources (say, food, water, and electricity). In other words, they use up some of the stuff you would otherwise be able to use only for yourse…

A better fitting analogy would be you ask FedEx to pick up a package at your house for delivery, and they have a new rule that they won't pick up packages unless they can look inside to make sure that they are packed correctly and do not contain anything that FedEx does not allow.

When they open and look into the package while in your house for the pickup, they are using your light to see and your heating/cooling is keeping the driver comfortable while they inspect the package.

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#329

Post by 0xy » Sat, Aug 14, 2021, 11:14 AM UTC

simondotau wrote: post #326

0xy wrote: post #324

Earlier quoted context omitted.

What you're missing is the statistical probability of two MD5 hashes colliding, which is astronomically unlikely. Your argument is essentially that a collision is more likely than a false positive, which would imply a false positive rate of 0.00% in NCMEC's database based on its size. It's clear that nothing will convince you if you believe that humans managing a database will never, ever make a mistake after over 30…

Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data. I am merely pushing back at your evidence-free claim that "it's full of false positives." Once again, you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources". You're reading far more into your sources than they are saying. And now you are conflating claims of false po…

>Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data

If NCMEC's databases have a false positive rate of 1 in 1,000, do you realise that means a false positive rate is substantially more likely than a hash collision?

>I am merely pushing back at your evidence-free claim that "it's full of false positives."

1 in 1,000 is "full of false positives" by my own standards, to which I've posted evidence from an employee whose company works directly with NCMEC.

>you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources".

NCMEC and law enforcement are one and the same. FBI employees work directly at NCMEC. [1] Law enforcement have direct access to the database, including for categorisation purposes. To suggest law enforcement's dataset is tainted yet NCMEC's is not doesn't make any sense to me.

>You are clearly very confused

Address the 1 in 1,000 claim. Thorn is an NCMEC partner and a Thorn employee has claimed a false positive rate of 1 in 1,000. In press releases, Thorn is even less sure at 1% failure rates. Thorn uses perceptual hashing.

I can't see how you can simultaneously claim the database isn't "full of false positives" while acknowledging a failure rate as abysmal as 1 in 1,000.

I also didn't conflate anything, both a hash collision and a perceptual hash collision are less likely than a false positive, by an extraordinary margin. Apple claims their algorithm has a collision 1 in 1,000,000,000,000 times. Compare that to 1 in 1,000.

The database is full of false positives, and now presumably you'll deny both industry claims and NCMEC partner claims.

[1] https://www.fbi.gov/investigate/violent-crime/cac

Re: Security Threat Model Review of the Apple Child Safety Features [pdf]

#330

Post by tzs » Sat, Aug 14, 2021, 11:21 AM UTC

zepto wrote: post #186

A key point that needs to be mentioned: we strongly dislike being distrusted. It might well be a genetic heritage. Being trusted in a tribe is crucial to survival, and so is likely wired deep into our social psychology. Apple is making a mistake by ignoring that. This isn’t about people not trusting Apple. It’s about people not feeling trusted by Apple. Because of this, it doesn’t matter how trustworthy the system is…

That's not at all clear.

A lot of people like strong border controls for instance, even if it means when they return to their country from abroad they have to go through more checks or present more documents to get in. Or consider large gated communities where you have to be checked by a guard to get in.

Many peoples seem fine with being distrusted as long as the distrust is part of a mechanism to weed out those who they feel really are not trustworthy and it is not too annoying for them to prove that they are not one of those people when they encounter a trust check.