ImageNet contains naturally occurring Apple NeuralHash collisions • Hacker News Board

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#211

Post by orhmeh09 » Thu, Aug 19, 2021, 7:01 PM UTC

tyingq wrote: post #202

orhmeh09 wrote: post #169

Earlier quoted context omitted.

Assuming you are American — where do you think your iCloud keys are stored? You do know Apple cooperates with US LE and intelligence? This is a nothing hamburger.

I concede that there are overlapping issues there. But if you're saying there aren't places China goes with this sort of info that's different from the US, I don't think any debate would change your mind.

So far I have no specific reason to think China goes places that are as deeply consequential and chilling than the US. What’s Assange up to these days?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#212

Post by tshaddox » Thu, Aug 19, 2021, 7:02 PM UTC

mrtranscendence wrote: post #151

stickfigure wrote: post #129

Earlier quoted context omitted.

Who said anything about fake porn hashes? China can just say to Apple: We want everyone whose phones contain XYZ subversive content. Don't like it? Don't sell phones. Apple will cave. Note that you don't even have to arrest everyone. The fear is enough to prevent thoughtcrime.

Agreed. If China can force Apple to do almost anything by threatening to ban iPhone sales, why bother with fake CSAM hashes? That just adds an extra step. It's not like the Chinese government needs to take pains to trick anyone about their attitude toward "subversive" material.

Exactly. Apple can already ship literally any conceivable software to iPhones. Do people really think their plan was to sneak functionality into this update and then update the CSAM database later, and they would have gotten away with it if it weren't for the brilliant privacy advocates pointing out that this CSAM database could be changed over time? That's pretty ludicrous. If the Chinese government wanted to (and thought it had sufficient leverage over Apple), they could literally just tell Apple to issue a software update that streams all desired private data to Chinese government servers.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#213

Post by fay59 » Thu, Aug 19, 2021, 7:02 PM UTC

e40 wrote: post #194

mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

> The end result is that some peon at Apple has to look at the images and mark them as not CSAM Btw, this reminded me of a podcast about FB's group to do just this. Because it negatively impacted the mental health of those FB employees, they farmed it out to the contractors in other countries. There were interviews with women in the Philippines, and it was having the same impact there.

It’s quite possible that you implied this, but I think that the true positives are the ones that had a mental health toll.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#214

Post by tablespoon » Thu, Aug 19, 2021, 7:03 PM UTC

stjohnswarts wrote: post #122

axelf4 wrote: post #116

Earlier quoted context omitted.

That is not a good comparison. The extra hashes would help China find out about more borderline citizens than it otherwise would have.

Have we established that a US NGO is accepting "CSAM" hashes from China or that they are cooperating with them at all? That seems unlikely and Apple hasn't yet announced plans with how they're going to scan phones in China, I mean wouldn't China just demand outright to have full scanning capabilities of anything on the phone since you don't have any protection at all from that in China?

> Have we established that a US NGO is accepting "CSAM" hashes from China or that they are cooperating with them at all?

I believe Apple's intention is to accept hashes from all governments, not just one US organization. One of their ineffectual concessions to the criticism was to require two governments provide the same hash before they'd start using it.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#215

Post by batch12 » Thu, Aug 19, 2021, 7:04 PM UTC

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

From the post yesterday discussing collisions, it doesn't seem outside the realm of possibility to take an image of CSAM and modify it until it has the hash that matches another target wanted either.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#216

Post by nomel » Thu, Aug 19, 2021, 7:05 PM UTC

mrtranscendence wrote: post #179

saithound wrote: post #163

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino…

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

Why is this question being downvoted? I too would like to know what this attack achieves.

From what I see, the end result of false flagging is either someone has CSAM in iCloud and you push them over the threshold that results in reporting and prosecution, or there is no CASM, so the reviewer sees all of the hash collision images, including those that are natural.

Is the problem that an attacker can force natural hash collision images to be viewed by a reviewer, violating that persons privacy? Do we know if this process is different than how Google, Facebook, Snapchat, Dropbox, Microsoft, and others have implemented these necessarily fuzzy matches for their CSAM scans of cloud hosted?

Or am I missing something that the downvoters saw?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#217

Post by tgsovlerkhgsel » Thu, Aug 19, 2021, 7:05 PM UTC

mrtranscendence wrote: post #182

tgsovlerkhgsel wrote: post #141

> Apple now has over 1.5 billion users so we are talking about a large pool of users at stake which increases the likelihood of even a low probability event manifesting This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their l…

> This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined. No one will have their lives ruined. If there's a false collision, someone at Apple has to look at the images as a final safeguard. If it's not actually C…

What probability do you ascribe to that reviewer clicking the wrong button, be it out of habit/zoning out (because the system usually shows them true positives), cheating (always clicking "yes" because it's usually correct and allows them to get paid without having to look at horrible images all day), mistake, wrong instructions (e.g. thinking that all images of children, or all porn including adult porn, should be flagged), confusion, or malice?

1 in 10? 1 in 100? 1 in 1000? Pick a number, you now have an estimate of how much room for error there is in the whole system.

If you consider 0.15 lives ruined on average per year acceptable, and the reviewers have a 1-in-1000 error rate, then the rest of the system has to make less than 1-in-10-million mistakes per year. And I'm pretty sure 1-in-1000 for the reviewers is very, very optimistic, even if you do quality checks, control for fatigue, etc.

On the other hand, hopefully there are some safeguards after the reviewers (e.g. human prosecutors who don't blindly rubber stamp). But my point is: The room for error in anything done at scale that has severe consequences is extremely small.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#218

Post by shuckles » Thu, Aug 19, 2021, 7:05 PM UTC

stjohnswarts wrote: post #140

endisneigh wrote: post #85

Earlier quoted context omitted.

> But more philosophically, it's your own device being turned against you to check you for criminal behavior. That's very different from somebody else checking up on you after you willingly interact with them. This literally only works once you willing send photos to iCloud.

For now. There is no technical hurdle preventing them from scanning everything locally and reporting back.

There wasn’t such a hurdle before or in the counterfactual where they built infrastructure to scan iCloud while also keeping iCloud Backups for every device.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#219

Post by tshaddox » Thu, Aug 19, 2021, 7:05 PM UTC

fsflover wrote: post #172

tshaddox wrote: post #158

Earlier quoted context omitted.

If the CCP says “put this arbitrary software into your next iPhone software update or we will halt all iPhone sales in China,” what do you think Apple is going to do? Isn’t the answer to both questions the same?

The answers are probably the same. What different is how hard it is to discover.

It apparently wasn't hard to "discover" the fact that this CSAM database can and will change over time. In fact, Apple explained this in detail as well as how they are attempting to avoid the problem of governments abusing the system. Are you suggesting that a different software update might be even easier to discover?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#220

Post by mrtranscendence » Thu, Aug 19, 2021, 7:05 PM UTC

stickfigure wrote: post #154

dylan604 wrote: post #57

Earlier quoted context omitted.

How does this fit within what Apple has stated will be the operating procedure for this scanning?

I don't know. But if the answer is "we won't snoop your browser cache", then people will quickly learn to store offending material under browser caches. Then Apple will have to start scanning browser caches. There is no policy fix for this; pedophiles aren't stupid.

Absolutely no one involved -- no one who conceived of this effort, no one who implemented it, and no one who defends it -- was under the impression that you couldn't get around this by not storing your images on iCloud. The idea is that it will catch pedophiles who aren't careful or tech-savvy enough, and such people exist, trust me. I suspect pedophiles aren't especially smarter than average, and most normal people aren't even going to be aware something like this exists.