Hash collision in Apple NeuralHash model • Hacker News Board

Re: Hash collision in Apple NeuralHash model

#181

Post by Edd314159 » Wed, Aug 18, 2021, 12:56 PM UTC

bo1024 wrote: post #152

Edd314159 wrote: post #105

I think I am in dire need of some education here and so I have questions: * Is this a problem with Apple's CSAM discriminator engine or with the fact that it's happening on-device? * Would this attack not be possible if scanning was instead happening in the cloud, using the same model? * Are other services (Google Photos, Facebook, etc.) that store photos in the cloud not doing something similar to uploaded photos, w…

Not complete answers but background: apple’s system works by having your device create a hash of each image you have. The hash (a short hexadecimal string) is compared to a list of known CP image hashes, and if it matches, then your image is uploaded to Apple for further investigation. A devastating scenario for such a system is if an attacker knows how to look at a hash and generate some image that matches the hash,…

> A devastating scenario for such a system is if an attacker knows how to look at a hash and generate some image that matches the hash, allowing them to trigger false positives any time.

This is my understanding too. But is this not also true for other (cloud-based) CSAM scanning systems? Why is Apple's special in this regard?

Re: Hash collision in Apple NeuralHash model

#182

Post by arsome » Wed, Aug 18, 2021, 12:57 PM UTC

avsteele wrote: post #128

Apple claimed 'one in a trillion' chance of a collision. This is a great example of why you should not trust such an assessment.

To be fair, log_2(1 trillion) is only about 40 bits.

Re: Hash collision in Apple NeuralHash model

#183

Post by bugmen0t » Wed, Aug 18, 2021, 12:57 PM UTC

Note that a hash collision is not the same as a pre-image attack. Though it seems to me that finding the latter is also feasible.

Re: Hash collision in Apple NeuralHash model

#184

Post by mannerheim » Wed, Aug 18, 2021, 12:58 PM UTC

shapefrog wrote: post #120

lifthrasiir wrote: post #10

Earlier quoted context omitted.

But you can essentially perform DoS attack to human checkers, effectively rendering the entire system grind to a halt. The entire system is too reliant on the performance of NeuralHash which can be defaced in many ways. [1] (Added later:) I should note that the DoS attack is only possible with the preimage attack and not the second preimage attack as the issue seemingly suggests, because you need the original CSAM to…

I have seen it suggested that everyone should flood the system with flagged images to overwhelm it in protest to this move by apple. Sounds pretty stupid to me to fill your phone with kiddie porn in protest, but you do you internet people.

You don't need to do that, just use images that collide with the hashes.

Re: Hash collision in Apple NeuralHash model

#185

Post by topynate » Wed, Aug 18, 2021, 12:58 PM UTC

Second preimage attacks are trivial because of how the algorithm works. The image goes through a neural network (one to which everyone has access), the output vector is put through a linear transformation, and that vector is binarized, then cryptographically hashed. It's trivial to perturb any image you might wish so as to be close to the original output vector. This will result in it having the same binarization, hence the same hash. I believe the neural network is a pretty conventional convolutional one, so adversarial perturbations will exist that are invisible to the naked eye.

This is useful for two purposes I can think of. One, you can randomize all the vectors on all of your images. Two, you can make problems for others by giving them harmless-looking images that have been cooked to give particular hashes. I'm not sure how bad those problems would be – at some point a police officer does have to look at the image in order to get probable cause. Perhaps it could lead to your Apple account being suspended, however.

Re: Hash collision in Apple NeuralHash model

#186

Post by philote » Wed, Aug 18, 2021, 12:58 PM UTC

varajelle wrote: post #75

vermilingua wrote: post #54

Earlier quoted context omitted.

iMessage no, WA by default yes, but can be disabled.

So no need of hash collisions then. One can simply directly send the child porn images to that person via WhatsApp and send her to jail.

But then you're searching and finding child porn images to send.

Re: Hash collision in Apple NeuralHash model

#187

Post by JimBard2 » Wed, Aug 18, 2021, 12:58 PM UTC

fsloth wrote: post #70

FabHK wrote: post #18

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6.…

"How can you use it for targeted attacks?" Just insert a known CSAM image on target's device. Done. I presume this could be used against a rival political party to ruin their reputation - insert bunch of CSAM images on their devices. "Party X is revealed as an abuse ring". This goes oh-so-very-nicely with Qanon conspiracy theories which even don't require any evidence to propagate widely. Wait for Apple to find the i…

Indeed this is not new and has probably been happening for many years already. There are services advertising on dark net markets to “ruin someone’s life” which means you pay some Ukrainian guy $300 in Bitcoin and he plants CSAM on a target’s computer.

Re: Hash collision in Apple NeuralHash model

#188

Post by CrimsonRain » Wed, Aug 18, 2021, 12:59 PM UTC

zaptheimpaler wrote: post #32

This is so overblown. Scanning images for CSAM seems to be a requirement followed by Facebook, Google, Insta and Snap already [1]: > To put this in perspective, in 2019 Facebook reported 65 million instances of CSAM on its platform, according to The New York Times. Google reported 3.5 million photos and videos, while Twitter and Snap reported “more than 100,000,” Apple, on the other hand, reported 3,000 photos. ALL o…

You're wrong. Scanning is NOT required by US law. The law says IF [1] you know about CSAM, then you MUST report. If you don't know, you don't have to report. And it is not your legal duty scan. Law even has privacy sections [2]. But if you DO scan, then you must report. So all these CSAM scan is bullshit. Companies can stop scanning if they want to.

[1] 18 US Code 2258A

[2] 18 US Code 2258A -> F.1, F.2, F.3

Re: Hash collision in Apple NeuralHash model

#189

Post by st_goliath » Wed, Aug 18, 2021, 1:02 PM UTC

FabHK wrote: post #18

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6.…

> 7. Apple reviewer....

This part IMO makes Apple itself the most likely "target", but for a different kind of attack.

Just wait until someone who wasn't supposed to, somewhere, somehow gets their hands on some of the actual hashes (IMO bound to happen eventually). Also remember that with Apple, we now have an oracle that can tell us. And with all the media attention around the issue, this might further incentivize people to try.

From that I can picture a chain of events something like this:

1. Somebody writes a script that generates pre-image collisions like in the post, but for actual hashes Apple uses.

2. The script ends up on the Internet. News reporting picks it up and it spreads around a little. This also means trolls get their hands on it.

3. Tons of colliding image are created by people all over the planet and sent around to even more people. Not for targeted attacks, but simply for the lulz.

4. Newer scripts show up eventually, e.g. for perturbing existing images or similar stunts. More news reporting follows, accelerating the effect and possibly also spreading perturbed images around themselves. Perturbed images (cat pictures, animated gifs, etc...) get uploaded to places like 9gag, reaching large audiences.

5. Repeat steps 1-4 until the Internet and the news grow bored with it.

During that entire process, potentially each of those images that ends up on an iDevice will have to be manually reviewed...

Re: Hash collision in Apple NeuralHash model

#190

Post by bitneuker » Wed, Aug 18, 2021, 1:03 PM UTC

magpi3 wrote: post #159

Can someone ELI5? I understand that a person can now generate an image with the same hash as an illegal image (such as child porn), but I don't understand how they can get it on someone's phone and I don't understand why someone would get in trouble for an image, when finally examined, that is clearly not child pornography.

A vulnerability by itself is not that dangerous, but in combination with a sophisticated attack, or another vulnerability can be disastrous. State actors have the resources to exploit a number of unknown bugs in combination with this collision to have Apple's systems flag persons of interest.

This, combined with human error during the manual review process might result in someone getting reported. Seeing as twitter (and other social media sites) jump on the bandwagon whenever someone gets accused of being a pedophile, this might destroy someones life.

The entire story might seem a bit to far fetched, but based on past events, you never know how bad something 'simple' as a hash collision can be.