Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hash collision in Apple NeuralHash model

github.com

Post reply on HN
Print view View original

671–680 of 725 posts

Re: Hash collision in Apple NeuralHash model

#671

Post by int_19h »

SXX wrote: post #372

Some people here in comments believe that whoever gonna check reported material on Apple side will never ever flag false-positive. We already know that NCMEC database itself don't exclusively contain child porn, but also some other photos that closely related ot CSAM. Even if those photos don't have actual CSAM on them. But let's ignore this fact. Do people who believe in behevolent Apple understand that CSAM don't a…

Quite frankly, I don't even care whether the reviewers are 100% perfect or not. I did not consent to having some random stranger out there reviewing my private photos. Even if it doesn't actually have any legal consequences, it's still unacceptable.

Re: Hash collision in Apple NeuralHash model

#672

Post by gitgud »

Edd314159 wrote: post #105

I think I am in dire need of some education here and so I have questions: * Is this a problem with Apple's CSAM discriminator engine or with the fact that it's happening on-device? * Would this attack not be possible if scanning was instead happening in the cloud, using the same model? * Are other services (Google Photos, Facebook, etc.) that store photos in the cloud not doing something similar to uploaded photos, w…

Basically you are right. There's nothing that special about hash collisions with image recognition.

I think this is blowing up because it's cathartic to see a technology you disagree with get undermined and basically broken by the community...

Re: Hash collision in Apple NeuralHash model

#673

Post by brokenmachine »

spoonjim wrote: post #594
sitharus wrote: post #56

Earlier quoted context omitted.

So the person would have to accept and save an image that when looks enough like CSAM to confuse a reviewer…

I don't see how a perfectly legal and normal explicit photograph of someone's 20-year-old wife would be indistinguishable to an Apple reviewer from CSAM, especially since some people look much younger or much older than their chronological age. So first, there would be the horrendous breach of privacy for an Apple goon to be looking at this picture in the first place, which the person in the photograph never consente…

The personal photo is unlikely to match a photo in the CSAM database though, or at least that's what is claimed by Apple with no way to verify if it's true or not.

Re: Hash collision in Apple NeuralHash model

#674

Post by brokenmachine »

int_19h wrote: post #669
mannerheim wrote: post #89

Earlier quoted context omitted.

No, it's important to point that out lest people think collisions can only be generated with contrived examples. I haven't studied neural hashes in particular, but for CNNs it's extremely trivial to come up with adversarial examples for arbitrary images. Anyway, as for human reviewers, depends on what the image being perturbed is. Computer repair employees have called the police on people who've had pictures of their…

Which would still be a privacy violation, since an actual human is looking at a photo you haven't consented to share with them.

That will be clearly laid out on page 1174 of Apples ToS that you had to click to be able to use your $1200 phone for anything but a paperweight.

Re: Hash collision in Apple NeuralHash model

#675

Post by brokenmachine »

Nextgrid wrote: post #47
FabHK wrote: post #18

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6.…

> the NeuralHashes of which, unless I'm mistaken, are not available Given the scanning is client-side wouldn't the client need a list of those hashes to check against? If so it's just a matter of time before those are extracted and used in these attacks.

I think there's some crypto mumbo-jumbo to make it so you can't know if an image matched or not.

Re: Hash collision in Apple NeuralHash model

#676

Post by faeriechangling »

JKCalhoun wrote: post #332
fsloth wrote: post #70

Earlier quoted context omitted.

"How can you use it for targeted attacks?" Just insert a known CSAM image on target's device. Done. I presume this could be used against a rival political party to ruin their reputation - insert bunch of CSAM images on their devices. "Party X is revealed as an abuse ring". This goes oh-so-very-nicely with Qanon conspiracy theories which even don't require any evidence to propagate widely. Wait for Apple to find the i…

> Just insert a known CSAM image on target's device. Or maybe thirty. You have to surpass the threshold. Also, if Twitter, Google, Microsoft are already deploying CSAM scanning in their services .... why are we not hearing about all the "swatting"?

Their implementations are not on-device and thus it's actually significantly more difficult to reverse engineer. Apples unique implementation of on-device scanning is much easier to reverse engineer and thus exploit.

Now apple is in this crappy situation where they can't claim their software is secure because it's open source and auditable, but they also can't claim it's secure because it's closed source and they fixed the problems in some later version because this entire debacle has likely destroyed all faith in their competence. If apple is in the position of having to boast "Trust us bro, your iPhone won't be exploited to get you SWATTED over CSAM anymore, we patched it" the big question is why is apple voluntarily adding something to their devices where the failure mode is violent imprisonment and severe loss of reputation when they are not completely competent?

This entire debacle reminds me of this video: https://www.youtube.com/watch?v=tVq1wgIN62E

Re: Hash collision in Apple NeuralHash model

#677

Post by brokenmachine »

jasamer wrote: post #465
sandworm101 wrote: post #225

Earlier quoted context omitted.

That is an Apple current policy, not any sort of law. If the FBI wants to know if there are any collisions on a particular phone, that will be shared. If the FBI wants to know of all single collisions, that too must be shared. A corporation's internal policy decisions are nothing when faced with a government official carrying a warrant. If they have data indicating possible crimes, no matter how small, they can be fo…

I thought the system is designed such that the threshold must be met to check the matches, i.e. it’s not possible to follow the FBIs request without changing the entire system.

int matchThreshold = 30;

if (numMatches > matchThreshold) { snitch(customer) && customer.life.ruin() }

Re: Hash collision in Apple NeuralHash model

#678

Post by faeriechangling »

FabHK wrote: post #61
michaelt wrote: post #50

Earlier quoted context omitted.

Just yesterday, here on HN there was an article [1] about adversarial attacks that could make road signs get misread by ML recognition systems I'd be astonished if it wasn't possible to do the same thing here. [1] https://news.ycombinator.com/item?id=28204077

But the remarkable thing there (and with all other adversarial attacks I've seen) is that the ML classifier is fooled, while for us humans it is obvious that it is still the original image (if maybe slightly perturbed). But in the case of Apple's CSAM detection, the collision would first have to fool the victim into seeing an innocent picture and storing it (presumably, they would not accept and store actual CSAM [^]…

Yes but the reviewers are not going to be viewing the original image, they are going to be viewing a 100x100 greyscale.

>If the premise is that the "innocent victim" would accept CSAM, then you might as well just send CSAM as an unscrupulous attacker.

This adds trojan horses embedded in .jpg files as an attack vector, which while maybe not overly practical, I could certainly imagine some malicious troll uploading "CSAM" to some pornsite.

Re: Hash collision in Apple NeuralHash model

#680

Post by IncRnd »

zepto wrote: post #391
avianlyric wrote: post #119

Earlier quoted context omitted.

> Also, defence by producing the original innocuous image At this point you’re already inside the guts of the justice system, and have been accused of distributing CSAM. Indeed depending on how diligent the prosecutor is, you might need to wait till trial before you can defend yourself. At that point you’re life as you know is already fucked. The only thing proving your innocence (and the need to do so is itself a co…

This isn’t true at all. If the creation of fakes is as easy as claimed, Neuralhash evidence alone will become inadmissible. There are plenty of lawyers and money waiting to establish this.

> This isn’t true at all.

> If the creation of fakes is as easy as claimed, Neuralhash evidence alone will become inadmissible.

Okay. https://github.com/anishathalye/neural-hash-collider

Post reply on HN