Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hash collision in Apple NeuralHash model

github.com

Post reply on HN
Print view View original

441–450 of 725 posts

Re: Hash collision in Apple NeuralHash model

#441

Post by shapefrog »

falcolas wrote: post #436
shapefrog wrote: post #431

Earlier quoted context omitted.

Sure, there are always cases - but was their photo of a grey blob that matched a hash but is clearly a grey blob or of a naked child? If the photo was a grey blob and they had to go through a judicial review for someone to look at the photo and confirm 'yes that is a grey blob' then color me wrong.

I'd view a "grey blob" to be the MVP of hash collisions. I doubt that this will end with grey blobs - I see it ending with common images (memes would be great for this) being invisibly altered to collide with a CSAM hash.

If you need a judicial review to confim that a slightly altered Bernie in Coat and Gloves meme is not the same image as the picture of a child being raped that they have on file then we have way bigger problems.

Re: Hash collision in Apple NeuralHash model

#442

Post by colejohnson66 »

short_sells_poo wrote: post #385
sweetheart wrote: post #370

Earlier quoted context omitted.

Their point is that the attack vector being described isn’t new, as CSAM could already be weaponized against folks, and we never really ever hear if that happening. So the OP is simply saying that perhaps it’s not an issue we need to worry about. I happen to agree with them.

So in your mind, because so far we've seen no evidence that this has been abused, it's nothing to worry about going forward? And that making an existing situation even more widespread is also completely OK?

It’s that the situation can’t be a “slippery slope” if there’s no evidence of there being one prior

Re: Hash collision in Apple NeuralHash model

#443

Post by moe »

snowwrestler wrote: post #381
sandworm101 wrote: post #363

Earlier quoted context omitted.

They are scanning images on online accounts, stuff that is stored on their systems and system files on machines (files they own). They are not, from what I have read, scanning customer-owned images stored on customer-owned devices.

Every AV (antivirus) software system scans customer-owned files on customer-owned devices. So the question is the same: if it’s easy for law enforcement to deputize such a system, where is the flood of cases built off of evidence gathered this way?

I feel like you are both chasing red herrings.

This is not about child porn nor about what AV software can or cannot do.

It's about normalising mass surveillance and implementing populace control. They don't want another Snowden to scare the public with reports about "backdoors" and mass privacy violations.

They want the coming generation to perceive it as normal. Because all phones do it, hasn't it always been this way, and think of the children.

Oh, these dissidents in $distant_country that will be muted and killed using Apple's shiny surveillance tech? Well, evil governments do what they do. But we are not like that. Over here it is only about the child predators. Trust us.

Apple has been an opponent of these developments for decades. Now they are spearheading it.

Re: Hash collision in Apple NeuralHash model

#444

Post by lifthrasiir »

robertoandred wrote: post #360
falcolas wrote: post #358

Earlier quoted context omitted.

Because the algorithm and list will be on your phone, and can (has, per TFA) be extracted.

You cannot extract or reverse the CSAM hashes. They've been encrypted and blinded using server-side-only keys. If TFA said that, it's lying.

After more reading of the whitepaper I think you are right. As I understand it, given the image hash H0 and CSAM hashes H[1]...H[n] (some might be duplicates in disguise) the algorithm proceeds like this:

- The device generates a secret X and divides it into X[1]...X[m] with the secret sharing algorithm. m is some large number and any k (but no less) copies out of X[i] are enough to reconstruct X.

- The device stores blinded hashes f(H[1])...f(H[n]). The function f itself is not known to the client.

- The image hash H0 is compressed with another function g to the range between 1 and n.

- The downscaled image data (for the human check) is encrypted with X and appended with (probably) random X[i].

- The result is then encrypted again with a key derived from f(H0) and sent to the server with an associated data g(H0).

- The server tries to decrypt it with a key derived from f(H[g(H0)]). This is only possible when H[g(H0)] = H0, i.e. H0 represents some known CSAM.

- You can only decrypt the second layer with at least k copies of X[i] then.

At this stage Apple can still learn the number of CSAM images less than k. The fix is described in an overly technical document and I can't exactly follow, but supposedly the client can inject an appropriate amount of synthetic data where only the first layer can be always decrypted and the second layer is bogus (including the presumed X[i]).

---

Assuming this scheme is correctly implemented, the only attack I can imagine is the timing attack. As I understand a malicious client can choose not to send false data. This will affect the number of items that pass the first layer of encryption, so the client can possibly learn the number of actual matches by adjusting the number of synthetic data since the server can only proceed to the next step with at least k such items.

This attack seems technically possible, but is probably infeasible to perform (remember that we already need 2^95 oracle operations, which is only vaguely possible even in the local device). Maybe the technical report actually has a solution for this, but for now I can only guess.

Re: Hash collision in Apple NeuralHash model

#445

Post by jug »

robertoandred wrote: post #340
HumblyTossed wrote: post #271

This can also be used to make Apple's system useless, no? If enough (millions?) of people were to, say, go to a web site and save generated gray-blobs to their phones, it would create enough false-positives to kill this system, right? Maybe game it and have everyone convert their various profile pics to these images.

And how would anyone know that those gray blobs match child porn?

Exactly, that’s the scary abuse scenario where people could send a whole load of this stuff to e.g a political enemy.

Re: Hash collision in Apple NeuralHash model

#446

Post by yeldarb »

I was curious how big of a deal this would be in a real-world attack (eg could someone use this to DDoS Apple's human reviewers?) so I ran the colliding images through another feature extraction model, OpenAI's CLIP, to see if they also fooled it.

They don't; I think it'd be much harder to create an image that both matches the NeuralHash of a CSAM image and also fools a generic model like CLIP as a sanity check for being a CSAM image.

I wrote up my findings here: https://blog.roboflow.com/apples-csam-neuralhash-collision/

Re: Hash collision in Apple NeuralHash model

#447

Post by bengale »

teitoklien wrote: post #432
bengale wrote: post #419

I don't understand why this is a concern. Someone would need 30 or so child porn images to generate these. Then they'd need to create these grey blobs and send them to the target. The target would need to save them in their photo library for some reason so they sync to iCloud. Then when all this triggers the review they'll see they are grey blobs and nothing else will happen?

Its astonishing , How society went from keeping personal photos private , to thinking its ok , to let a random apple employee scan their photos for suspicion and debate on how human reviewers from a faceless corporation can easily stop these mistakes there. I’d trust the algorithm more than the human reviewing it , considering the algorithm already shows flaws and loopholes , the human part of it does not bring any c…

But they won't be my private photos, the only way it gets to the point where a human see's them is that I have a bunch of child porn uploaded into iCloud, or someone puts a bunch of these grey blob images into my library. Neither of those are my images. There is no chance 30+ of my personal images have hash collisions with this database of child porn.

> How society went from keeping personal photos private

Also, didn't the local photo development shops used to call the police quite frequently when dodgy images were sent in to be developed? I remember it happening once at our local supermarket.

Re: Hash collision in Apple NeuralHash model

#448

Post by stetrain »

short_sells_poo wrote: post #408
robertoandred wrote: post #331

Earlier quoted context omitted.

Or rather, it will share some gray blob apparently.

What is the point of sharing a gray blob? How is that going to prove anything?

I think the gray blob refers to engineered hash collisions, like the example in the article link.

Re: Hash collision in Apple NeuralHash model

#449

Post by robertoandred »

mannerheim wrote: post #400
robertoandred wrote: post #335

Earlier quoted context omitted.

How will you know something collides?

> Apple’s method of detecting known CSAM is designed with user privacy in mind. Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child-safety organizations. Apple further transforms this database into an unreadable set of hashes, which is securely stored on users’ devices. https://www.apple.com/child-safety/pdf/CSAM…

Yeah, so how would you know something would collide with the hash?

Re: Hash collision in Apple NeuralHash model

#450

Post by teitoklien »

bengale wrote: post #447
teitoklien wrote: post #432

Earlier quoted context omitted.

Its astonishing , How society went from keeping personal photos private , to thinking its ok , to let a random apple employee scan their photos for suspicion and debate on how human reviewers from a faceless corporation can easily stop these mistakes there. I’d trust the algorithm more than the human reviewing it , considering the algorithm already shows flaws and loopholes , the human part of it does not bring any c…

But they won't be my private photos, the only way it gets to the point where a human see's them is that I have a bunch of child porn uploaded into iCloud, or someone puts a bunch of these grey blob images into my library. Neither of those are my images. There is no chance 30+ of my personal images have hash collisions with this database of child porn. > How society went from keeping personal photos private Also, didn…

Hash collisions dont need grey images You can take a perfectly normal image , play around with a range of its individual pixels, to get collisions too.

So someone might forward you your personal pics from your meeting with them and its pixels might get edited by the messenger app you use to save the picture into your photos to specifically collide its hash with a csam image, while to you the image will look perfectly normal

This is one example , lets say you 100% trust every developer of every app that you download on apple’s phone.

Fine, but that’s already a lot of trust on a lot of people about something that can ruin your life.

Do you now trust every single image which might get auto downloaded to your gallery by a malicious actor , what if a random anon person messages you with 100 such colliding images , enough to cross threshold to get the authorities knocking your door ,

Is this “feature” really worth it , to have the inconvenience of authorities knocking your door and going through legal trouble ?

For an iphone ?

Post reply on HN