Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

251–260 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#251

Post by mrtranscendence »

saithound wrote: post #232
mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

Okay, let's play peon. Here are three perfectly legal and work-safe thumbnails of a famous singer: https://imgur.com/a/j40fMex . The singer is underage in precisely one of the three photos. Can you decide which one? If your account has a large number of safety vouchers that trigger a CSAM match, then Apple will gather enough fragments to reassemble a secret key X (unique to your device) which they can use to decrypt…

Fair enough. I suppose it's true that you could create a colliding sexually explicit image where age is indeterminate, and the reviewer may not realize it isn't a match.

> Given the ability to produce hash collisions, an adversary can easily generate photos that fail this visual inspection as well.

Apple could easily fix this by also showing a low-res version of the CSAM image that was collided with, but I'll grant that they may not be able to do that legally (and reviewers probably don't want to look at actual CSAM).

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#252

Post by foerbert »

endisneigh wrote: post #85
foerbert wrote: post #79

Earlier quoted context omitted.

I don't think we can just appeal to the status quo here and assume it's acceptable. There's a couple reasons. First, how many people really understood this previously? Did society at large actually knowingly accept the current state of things, or did it just happen without most people realizing it? Even here on HN where we'd expect to find people way more knowledgeable about it than in general I'm not sure how well k…

> But more philosophically, it's your own device being turned against you to check you for criminal behavior. That's very different from somebody else checking up on you after you willingly interact with them. This literally only works once you willing send photos to iCloud.

Right. I mentioned that. It's still your own device doing it.

It's like announcing to your family member you're going to tell your neighbor you committed a crime and your family member turns you in first. Yeah, you could expect your neighbor to do the same, but are you really not going to feel any differently about the fact it was your family that turned you in?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#253

Post by c3534l »

breuleux wrote: post #67
stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Presumably Apple would be afraid that, say, the EU becomes suspicious, issues a court order to obtain the hashes, notices they cannot audit the CCP hashes, pointedly asks "what is this", becomes absolutely livid that their citizens are spied on by a country that is not them, fines Apple out the wazoo, then extradites whoever is responsible and puts them in prison. I mean, China's not the only player in this. Putting…

I thought it was already common knowledge that China puts in different hardware backdoors for computers destined to different countries. I remember a while back a news story where China accidentally shipped a box of phones backdoored for China into the US.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#254

Post by giantrobot »

mrtranscendence wrote: post #179
saithound wrote: post #163

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino…

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a legitimate CSAM image and modify it such that when scaled for use in the perceptual algorithm(s) it changes to be some politically sensitive image. Upon review it'll look like CSAM so off it goes to reporting agencies.

Because the perceptual hash algorithms are presented as black boxes the image they perceive isn't audited or reviewed. There's zero recognition of this weakness by Apple or NCMEC (and their equivalents). For the system to even begin to be trustworthy all content would need to be reviewed raw and scaled-as-fed-into-the-algorithm.

[0] https://bdtechtalks.com/2020/08/03/machine-learning-adversar...

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#255

Post by kmbfjr »

sennight wrote: post #136
endisneigh wrote: post #8

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’…

We are talking about how everyone who gave Apple money now has a potential probable cause vector that they didn't before. Everyone running the software is a suspect by default. Ask black Americans how they feel about setting the bar low for probable cause. "Following the 2004 Madrid train bombings, fingerprints on a bag containing detonating devices were found by Spanish authorities. The Spanish National Police share…

Reading about incidences such as this has made me think critically about all cloud services in the United States, and the conclusion is simply not to use them.

Sure, the probability is lower than getting struck by lightning. I certainly don't play in the rain and I won't be using cloud services where I'm exposed to this kind of nonsense with the FBI.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#256

Post by robertoandred »

saithound wrote: post #232
mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

Okay, let's play peon. Here are three perfectly legal and work-safe thumbnails of a famous singer: https://imgur.com/a/j40fMex . The singer is underage in precisely one of the three photos. Can you decide which one? If your account has a large number of safety vouchers that trigger a CSAM match, then Apple will gather enough fragments to reassemble a secret key X (unique to your device) which they can use to decrypt…

Apple can only ever see the visual derivatives in vouchers of images that match CSAM hashes, not vouchers of all your images.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#257

Post by tshaddox »

giantrobot wrote: post #228
tshaddox wrote: post #212

Earlier quoted context omitted.

Exactly. Apple can already ship literally any conceivable software to iPhones. Do people really think their plan was to sneak functionality into this update and then update the CSAM database later, and they would have gotten away with it if it weren't for the brilliant privacy advocates pointing out that this CSAM database could be changed over time? That's pretty ludicrous. If the Chinese government wanted to (and t…

> they could literally just tell Apple to issue a software update that streams all desired private data to Chinese government servers. Uh...this already happened. [0][1] [0] https://www.macrumors.com/2021/05/17/apple-security-compromi... [1] https://support.apple.com/en-us/HT208351

Not quite. Those are still ostensibly servers located in China but not directly controlled by the government (edit: apparently the hosting company is owned by Guizhou provincial government). But yes, this is precisely my point. Any slippery slope argument about Apple software on iPhones is equivalent to any conceivable slippery slope argument about Apple software on iPhones. If you're making one of these arguments, you're actually just arguing against Apple having the ability to issue software updates to iPhones (and by all means, make that argument!).

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#258

Post by foerbert »

mrtranscendence wrote: post #175
foerbert wrote: post #69

Earlier quoted context omitted.

The difference is the ease with which they can demur. Before, it would be a whole heck of a lot of new, additional work. They also have the problem of actually introducing it without being noticed, or having to come up with some cover for the new behavior. Now? Well now it's real simple. It will even conveniently not expose the actual images it's checking for. Apple now has significantly less ability to rationally re…

> The difference is the ease with which they can demur. If Apple can be cowed by China into adding fake CSAM hashes by threat of banning iPhone sales, they could be cowed to surveil Chinese citizens in the search for subversive material. It's no skin off China's back if it's harder for Apple -- they'll either make the demand or they won't. This changes basically nothing.

I think that's a too simplistic view.

It's kinda true, but ignores how humans really work. Apple will be pushed around to a degree, but there will be limits. The harder the ask now the less China can ask later. And the more Apple can protest about the difficulty and impossibility and other consequences they will face, the more likely China is to back off.

Both sides want to have their cake and eat it too, and will compromise to make it basically work. But if China makes demands so excessive they get Apple to cut ties, China loses. Apple has the money, demand, customer loyalty, and clout to make things real uncomfortable. Apple would have to pay a hefty price, but if any company can do it... it's them.

So I don't think it's fair to say that no matter what China will just demand whatever whims strike it each day and everybody will play ball or gtfo. That just isn't how shit works.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#259

Post by robertoandred »

tgsovlerkhgsel wrote: post #217
mrtranscendence wrote: post #182

Earlier quoted context omitted.

> This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined. No one will have their lives ruined. If there's a false collision, someone at Apple has to look at the images as a final safeguard. If it's not actually C…

What probability do you ascribe to that reviewer clicking the wrong button, be it out of habit/zoning out (because the system usually shows them true positives), cheating (always clicking "yes" because it's usually correct and allows them to get paid without having to look at horrible images all day), mistake, wrong instructions (e.g. thinking that all images of children, or all porn including adult porn, should be f…

Even if Apple’s manual review fails, there’s still the NCMEC’s review. There are several layers before anything goes to law enforcement.
Post reply on HN