Hash collision in Apple NeuralHash model • Hacker News Board

Re: Hash collision in Apple NeuralHash model

#301

Post by merpnderp » Wed, Aug 18, 2021, 2:22 PM UTC

sandworm101 wrote: post #215

Grustaf wrote: post #200

Earlier quoted context omitted.

How likely is it that you will have enough colliding images in your photo library to even trigger a review? I'm guessing you need at least 5 images, perhaps much more, to trigger it. In any case, 1 image is definitely not enough.

For a normal random person, a grey man. In the real world, a single collision could be enough for the police to acquire further access to people they are already looking at. If the police want access to your phone for other reasons (drugs, taxes, illegal speech) they can use that one collision to get a warrant which will give them greater access. It is akin to cops wanting to search a car. They don't need a warrant.…

Isn't Apple seeding their database with fake hits so that a court can't order them to turn over everyone who's just a few shy of Apple's threshold? So when this database leaks, won't there be a lot of people accused of horrific crimes simply because Apple seeded their database with random hits?

Why would people pay a company to falsely accuse them of horrific crimes?

Re: Hash collision in Apple NeuralHash model

#302

Post by endisneigh » Wed, Aug 18, 2021, 2:23 PM UTC

kemayo wrote: post #263

endisneigh wrote: post #218

Earlier quoted context omitted.

They can see the image - why would they import a random image into their library from someone they don’t know?

WhatsApp has a really weird default behavior: it imports all images you're sent into your photo library. This is a smart thing to disable, even outside this recent discussion of CSAM. https://faq.whatsapp.com/android/how-to-stop-saving-whatsapp...

In that scenario this attack is unnecessary. Someone could just send you legitimate child pornography and then immediately tell the authorities that you possess said things.

Re: Hash collision in Apple NeuralHash model

#303

Post by floatingatoll » Wed, Aug 18, 2021, 2:24 PM UTC

HumblyTossed wrote: post #271

This can also be used to make Apple's system useless, no? If enough (millions?) of people were to, say, go to a web site and save generated gray-blobs to their phones, it would create enough false-positives to kill this system, right? Maybe game it and have everyone convert their various profile pics to these images.

No, because Apple will simply start killing Apple IDs and blocking hardware by serial number for abusive behavior towards Apple when y’all do that, and that’s a very expensive problem to overcome.

“You tried to exploit our production systems and we’re ending our customer relationship with you over it” is a classic Apple move and no one outside of the Hackintosh community realizes that their devices include crypto-signed attestations of their serial number during Apple service sign-ins.

Re: Hash collision in Apple NeuralHash model

#304

Post by kemayo » Wed, Aug 18, 2021, 2:25 PM UTC

endisneigh wrote: post #302

kemayo wrote: post #263

Earlier quoted context omitted.

WhatsApp has a really weird default behavior: it imports all images you're sent into your photo library. This is a smart thing to disable, even outside this recent discussion of CSAM. https://faq.whatsapp.com/android/how-to-stop-saving-whatsapp...

In that scenario this attack is unnecessary. Someone could just send you legitimate child pornography and then immediately tell the authorities that you possess said things.

Yup! It's a legitimately crazy default.

Edit: Though, to be fair, the specific hash-collision scenario would be that someone could send you something that doesn't look like CSAM and so you wouldn't reflexively delete it.

Re: Hash collision in Apple NeuralHash model

#305

Post by f3d46600-b66e » Wed, Aug 18, 2021, 2:26 PM UTC

FabHK wrote: post #18

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6.…

What makes CSAM database private?

It's my understanding that many tech companies (Microsoft? Dropbox? Google? Apple? Other?) (and many people in those companies) have access to the CSAM database, which essentially makes it public.

Re: Hash collision in Apple NeuralHash model

#306

Post by devwastaken » Wed, Aug 18, 2021, 2:26 PM UTC

hughrr wrote: post #7

That’s end game. Now you can use it for targeted attacks against innocent people. This needs to be shut down and disposed of immediately. There is no other outcome which is socially acceptable for Apple. I feel vindicated now. There are a lot of people saying that I’m insane as I’ve dumped the entire iOS ecosystem in the last week. But Craig was busy steamrolling out the marketing still only a couple of days back abo…

The collisions are supposedly reviewed, my concern is that the process for photodna isn't going to always be the same, and we don't actually have any knowledge as to whether their claims are true. Eventually they will phase out human intervention and replace with gameable AI. 3 letter agencies don't need to review the actual images, they can easily get federal warrants based on some numbers.

Apple is content with putting in backdoors. Theres collusion behind the scenes here, they know it's bad, but hey, it's apple, it's ran by manipulators and liars and forms it's own cult.

Re: Hash collision in Apple NeuralHash model

#307

Post by Bjartr » Wed, Aug 18, 2021, 2:26 PM UTC

bostonsre wrote: post #259

y7 wrote: post #65

Earlier quoted context omitted.

Cross-posting from another thread [1]: 1. Obtain known CSAM that is likely in the database and generate its NeuralHash. 2. Use an image-scaling attack [2] together with adversarial collisions to generate a perturbed image such that its NeuralHash is in the database and its image derivative looks like CSAM. A difference compared to server-side CSAM detection could be that they verify the entire image, and not just the…

Why would someone do that? Why not just send the original if both are flagged as the original?

Because having actual CSAM images is illegal.

Re: Hash collision in Apple NeuralHash model

#308

Post by endisneigh » Wed, Aug 18, 2021, 2:28 PM UTC

kemayo wrote: post #304

endisneigh wrote: post #302

Earlier quoted context omitted.

In that scenario this attack is unnecessary. Someone could just send you legitimate child pornography and then immediately tell the authorities that you possess said things.

Yup! It's a legitimately crazy default. Edit: Though, to be fair, the specific hash-collision scenario would be that someone could send you something that doesn't look like CSAM and so you wouldn't reflexively delete it.

If it doesn’t look like it wouldn’t a human reviewer disregard it once it gets to that point?

Personally I don’t really see the issue.

Re: Hash collision in Apple NeuralHash model

#309

Post by xpe » Wed, Aug 18, 2021, 2:28 PM UTC

coldtea wrote: post #135

d33 wrote: post #107

Earlier quoted context omitted.

In case of Appelbaum, are there any solid reasons to believe that the accusations are untrue? For Assange, I think that the victim admitted that the accusation was fabricated, isn't that the case?

Accusations must be proven true, not untrue by the accused. And besides the "victim" in the latter case there was a whole lot of diplomatic pressure and political commotion to set him up, with carrots and sticks and the aid of friendly satellite states.

> Accusations must be proven true, not untrue by the accused.

I respect this ethical claim; however, there is considerable variation in how legal systems operate in this domain.

Re: Hash collision in Apple NeuralHash model

#310

Post by Grustaf » Wed, Aug 18, 2021, 2:28 PM UTC

gorbachev wrote: post #288

Grustaf wrote: post #200

Earlier quoted context omitted.

How likely is it that you will have enough colliding images in your photo library to even trigger a review? I'm guessing you need at least 5 images, perhaps much more, to trigger it. In any case, 1 image is definitely not enough.

For individual users the risk is very small, but when you have a billion users the chances of innocent people being caught up is pretty much 100%. Platform owners like Google, Apple, Twitter and Facebook should really keep stuff like that in mind when they deploy algorithmic solutions like this. Like dhosek said, the manual review step should reduce the risk quite a bit, though.

They did keep it in mind, that's why they require 30 matches, which gets the false positive rate down to 1 in a trillion per year, so on average it will happen about 1 per millennium, given a billion users.

So that's per photo library, not per photo. The rate per photo in their testing was 3 in 100 million, then they added a 30x safety margin and assumed it's actually 1 in a million.

https://www.zdnet.com/article/apple-to-tune-csam-system-to-k...