ImageNet contains naturally occurring Apple NeuralHash collisions • Hacker News Board

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#141

Post by tgsovlerkhgsel » Thu, Aug 19, 2021, 6:29 PM UTC

> Apple now has over 1.5 billion users so we are talking about a large pool of users at stake which increases the likelihood of even a low probability event manifesting

This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined.

What's even worse, when those individuals are facing whatever they're facing, they'll have to argue against the one-in-a-billion odds. If there are jurisdictions where defense lawyers don't get access to the images their client is accused of, and prosecutors and judges don't look at the images but only at the surrounding evidence (which says this person is guilty, with 99.9999999% certainty), Apple may have built a system that's statistically expected to send an innocent person to prison.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#142

Post by tablespoon » Thu, Aug 19, 2021, 6:29 PM UTC

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

>> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse.

> If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do? Is anyone so naive that they believe the CCP wouldn't deliver such an ultimatum? Apple's position seems to completely ignore recent Chinese history.

Apple policy Edit: just thought of another way Apple's policy could be easily circumvented and therefore cannot be regarded as a serious proposal: get two countries to collaborate to add politically sensitive hashes to the list (e.g. China and North Korea, or China and Cambodia). That doesn't even require Apple to be coerced.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#143

Post by sigmar » Thu, Aug 19, 2021, 6:30 PM UTC

slg wrote: post #113

colpabar wrote: post #101

Earlier quoted context omitted.

> It is only moving from the cloud to the local device. But isn't that exactly why this is such a big deal? It sets a precedent that it's ok that devices are scanning your local device for digital contraband. Sure, right now it's only for photos that are going to be uploaded to iCloud anyway. But how long before it scans everything, and there's no way to opt out? I don't see this as so much a question of apple's trus…

>But how long before it scans everything, and there's no way to opt out? Do we think this is detectable? If yes, then why worry about it if we will know when this switch is made? If not, why did we trust Apple that this wasn't happening already? That is the primary thing I don't understand, this fear rests on an assumption that Apple is a combination of both honest and corrupted. If they are honest, we have no reason…

Are we going to need to reverse engineer every single Apple update to make sure the feature hasn't creeped into non-iCloud uses? Is the inevitable Samsung version of this system going to be as privacy-preserving? How are we sure the hash list isn't tainted? All of these problems are solved by one principle: Don't put the backdoor code in the OS to begin with.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#144

Post by breuleux » Thu, Aug 19, 2021, 6:31 PM UTC

dgellow wrote: post #121

breuleux wrote: post #67

Earlier quoted context omitted.

Presumably Apple would be afraid that, say, the EU becomes suspicious, issues a court order to obtain the hashes, notices they cannot audit the CCP hashes, pointedly asks "what is this", becomes absolutely livid that their citizens are spied on by a country that is not them, fines Apple out the wazoo, then extradites whoever is responsible and puts them in prison. I mean, China's not the only player in this. Putting…

I think that you overestimate the EU reaction. Every few years we learn that our Europeans leaders and some citizens have been again spied by foreign powers, such as the US, and absolutely nothing ever happened.

The US is an ally and it is somewhat harder to punish a nation state than a company. Why would Apple take the risk? China can't exactly reveal that they are banning an American company for not spying on American citizens, and it's not clear what convincing pretext they could provide instead, so I don't think they would actually go through with a ban and Apple would probably just call their bluff.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#145

Post by dont__panic » Thu, Aug 19, 2021, 6:31 PM UTC

nonbirithm wrote: post #93

dannyobrien wrote: post #64

Earlier quoted context omitted.

One of the things that is happening now is that the entire PhotoDNA system is finally coming under the level of oversight that it should have had right from the start. I can tell you from working in this area that it's possible for someone to have their lives ruined by a misplaced investigation, have that investigation abandoned because they turn out to be obviously innocent, and for that to not be well-known, becaus…

We certainly have Apple's PR blunder to thank for that, but if PhotoDNA always held that potential for abuse due to its very nature, why did we remain silent for 13 years? Maybe it's because Google and Microsoft and others' policy of security through obscurity actually succeeded in preventing the details of PhotoDNA from coming to light, and it took Apple exposing their hashing model to reverse engineering by includi…

Considering I didn't know about:

- PhotoDNA

- CSAM scanning on cloud photo platforms

- the acronym "CSAM"

Before this whole Apple client-side scanning debacle... seems pretty likely. A lot of privacy-focused people also avoid Google and Microsoft cloud services like the plague and trusted Apple up to this point to protect their privacy. The fact that Apple was (and is) scanning iCloud Photos libraries for CSAM unbeknownst to most of us is just another violation of that trust and shows just how far the "what happens on your iphone, stays on your iphone" privacy marketing extends (read: not past your iphone, and sometimes not even on your iphone).

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#146

Post by zimpenfish » Thu, Aug 19, 2021, 6:31 PM UTC

stickfigure wrote: post #120

endisneigh wrote: post #87

Earlier quoted context omitted.

I'm not talking about Apple, I'm saying in general technology has already been deployed to make what you're describing possible. So where's the evidence of abuse?

No vendors are snooping your phone/computer browser cache, so this attack vector does not yet exist. Apple is building it.

> No vendors are snooping your phone/computer browser cache

a) That you know of b) Apple won't be doing this either

> Apple is building it.

No, they aren't.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#147

Post by tablespoon » Thu, Aug 19, 2021, 6:32 PM UTC

burkaman wrote: post #117

yongjik wrote: post #110

Earlier quoted context omitted.

These scenarios sound rather like "the wrong side of airlock" stories[1]. Why would China go through an elaborate scheme with fake child-porn hashes, when it can already arrest these people on made-up charges, and simply tell Apple to provide the private key for their phones, so that they can read and insert whatever real/fake evidences they want? [1] I'm stealing the expression from this excellent article: https://d…

Because they don't know who to arrest yet. The idea isn't to fabricate a charge, it's to locate people sharing politically sensitive images that the government hasn't already identified.

> Because they don't know who to arrest yet. The idea isn't to fabricate a charge, it's to locate people sharing politically sensitive images that the government hasn't already identified.

And maybe even identify avenues for sharing that they haven't already identified and monitored/controlled (e.g. some encrypted chat app they haven't blocked yet).

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#148

Post by Y_Y » Thu, Aug 19, 2021, 6:33 PM UTC

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

What about giving a censored version of the appropriate image? Like put a big black rectangle covering whatever awful thing is the subject, and just (e.g.) show some feet and hands and a background.

Then you could provide a proof that an image which is the same as the "censored", one except for the masked part, has the perceptual hash specified. I don't know if this is technically feasible (but I'd be happy for someone knowledgeable to opine). I also admit that there are secondary concerns, like the possibility of recognising the background image, and this being used by someone to identify the location, or tipping off a criminal.

Probably it would only be appropriate to do this in the case of someone being accused, and maybe then in a way where they couldn't relay the information, since apparently they don't want to make the hash database public.

Also, for the record, I'm spitballing here about infosec. This isn't me volunteering to draw black boxes or be called by anyone's defense.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#149

Post by stjohnswarts » Thu, Aug 19, 2021, 6:33 PM UTC

rootusrootus wrote: post #32

minitoar wrote: post #11

Earlier quoted context omitted.

The damage is already done by the time it gets to the point of devices being confiscated.

By the time the FBI comes knocking for your devices, they have a lot of evidence, not a list of hash collisions.

I doubt it, all they need is this stuff and they can get a warrant to rummage through your stuff and take all your computers, usb drives, etc and also put your name on a watch list and your permanent record. Much like newspapers retracting mistakes if the story doesn't pan out, it goes on the back page. Plenty enough to wreck your life.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#150

Post by bequanna » Thu, Aug 19, 2021, 6:33 PM UTC

alfalfasprout wrote: post #2

There's been a lot of focus on the likelihood of collisions and whether someone could upload eg; an image with a matching hash to your device to "set you up", etc. But what's still extremely concerning is that there is still no guarantee that the hash list used can't be coopted for another purpose (eg; politically insensitive content).

> there is still no guarantee that the hash list used can't be coopted for another purpose (eg; politically insensitive content).

That isn't a bug, it is a feature and will be the main use of this functionality.

The "preventing child pornography" reasoning was specifically chosen so that Apple could openly coordinate with governments to violate your privacy while avoiding criticism.