Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hash collision in Apple NeuralHash model

github.com

Post reply on HN
Print view View original

61–70 of 725 posts

Re: Hash collision in Apple NeuralHash model

#61

Post by FabHK »

michaelt wrote: post #50
FabHK wrote: post #35

Earlier quoted context omitted.

Remains to be shown whether that is possible, though.

Just yesterday, here on HN there was an article [1] about adversarial attacks that could make road signs get misread by ML recognition systems I'd be astonished if it wasn't possible to do the same thing here. [1] https://news.ycombinator.com/item?id=28204077

But the remarkable thing there (and with all other adversarial attacks I've seen) is that the ML classifier is fooled, while for us humans it is obvious that it is still the original image (if maybe slightly perturbed).

But in the case of Apple's CSAM detection, the collision would first have to fool the victim into seeing an innocent picture and storing it (presumably, they would not accept and store actual CSAM [^]), then fool the NeuralHash into thinking it was CSAM (ok, maybe possible, though classifiers perceptual hash), then fool the human reviewer into also seeing CSAM (unlike the innocent victim).

[^] If the premise is that the "innocent victim" would accept CSAM, then you might as well just send CSAM as an unscrupulous attacker.

Re: Hash collision in Apple NeuralHash model

#62

Post by henrikeh »

hughrr wrote: post #51
henrikeh wrote: post #37

Earlier quoted context omitted.

Yes. That is called NCMEC in the US and it is a core aspect of how this whole process works. If you don’t understand the details of this, I’ll recommend this podcast episode which sums it up and discusses the implications https://atp.fm/443

I am aware of the details. The episode of Brass Eye comes into context here, the relevant clip being as follows, showing exactly the issues of competence. https://youtu.be/_U-7L1tmBAo

I’m sorry, what is your point? I legitimately don’t understand what you mean.

Re: Hash collision in Apple NeuralHash model

#63

Post by the8472 »

sitharus wrote: post #56
the8472 wrote: post #27

Earlier quoted context omitted.

> 7. Apple reviewer confuses a featureless blob of gray with CSAM material, several times A better collision won't be a grey blob, it'll take some photoshopped and downscaled picture of a kid and massage the least significant bits until it is a collision. https://openai.com/blog/adversarial-example-research/

So the person would have to accept and save an image that when looks enough like CSAM to confuse a reviewer…

Depending on what algorithm apple uses to generate the "sample" that's shown to the reviewer it may be possible to generate a large image that looks innocent unless downscaled with that specific algorithm and to a specific resolution

Re: Hash collision in Apple NeuralHash model

#64

Post by LeoNatan25 »

dijit wrote: post #40
bko wrote: post #20

Earlier quoted context omitted.

Isn't it weird how it is weaponized against political enemies but the one person everyone knows did engage in exploitation was protected for decades?

I don't think that's weird. Those people are not political enemies, they're allies for whatever regime and their support is important. Better to keep someone powerful in your pocket than to oust them and let someone uncorruptible or uncompromising assume the power vacuum.

No one is “ uncorruptible or uncompromising”. They just have to start looking from scratch.

Re: Hash collision in Apple NeuralHash model

#65

Post by y7 »

FabHK wrote: post #18

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6.…

Cross-posting from another thread [1]:

1. Obtain known CSAM that is likely in the database and generate its NeuralHash.

2. Use an image-scaling attack [2] together with adversarial collisions to generate a perturbed image such that its NeuralHash is in the database and its image derivative looks like CSAM.

A difference compared to server-side CSAM detection could be that they verify the entire image, and not just the image derivative, before notifying the authorities.

[1] https://news.ycombinator.com/item?id=28218922

[2] https://bdtechtalks.com/2020/08/03/machine-learning-adversar...

Re: Hash collision in Apple NeuralHash model

#66

Post by nyuszika7h »

WA wrote: post #53
zaptheimpaler wrote: post #32

This is so overblown. Scanning images for CSAM seems to be a requirement followed by Facebook, Google, Insta and Snap already [1]: > To put this in perspective, in 2019 Facebook reported 65 million instances of CSAM on its platform, according to The New York Times. Google reported 3.5 million photos and videos, while Twitter and Snap reported “more than 100,000,” Apple, on the other hand, reported 3,000 photos. ALL o…

Is it so hard to understand? Some people don’t use cloud storage for precisely the reason that the photos are not encrypted. Now they can’t even use their phone for storing photos. The thing with "only when iCloud is enabled" is only for now. It’s trivial to make Scanning all photos default in a future version.

That would require a software update and would definitely not go unnoticed. Would you rather they implement scanning on server side and never be able to enable end-to-end encryption for iCloud Photos? I imagine that might be the end goal, otherwise I don't see why they wouldn't have just done it on server side.

Sure, this system still has the potential to be abused, but if I had to choose between "end-to-end encrypted with local scanning before upload, which can maybe be abused but has a higher chance of being noticed if it is, and can be disabled with a jailbreak if you're really paranoid" and "not end-to-end encrypted, Apple can inspect my whole photo library whenever they desire, which can go completely unnoticed due to gag orders forcing them to hand data over to governments, not even requiring a software update, and you are completely powerless to do anything about it except for not using iCloud Photos at all", I would definitely choose the former.

Re: Hash collision in Apple NeuralHash model

#67

Post by dijit »

LeoNatan25 wrote: post #64
dijit wrote: post #40

Earlier quoted context omitted.

I don't think that's weird. Those people are not political enemies, they're allies for whatever regime and their support is important. Better to keep someone powerful in your pocket than to oust them and let someone uncorruptible or uncompromising assume the power vacuum.

No one is “ uncorruptible or uncompromising”. They just have to start looking from scratch.

There are many people who are incorruptible or uncompromising, we usually dislike them.

Maybe "not sympathetic" would prevent me from being nerd sniped. :)

Re: Hash collision in Apple NeuralHash model

#68

Post by formerly_proven »

sitharus wrote: post #56
the8472 wrote: post #27

Earlier quoted context omitted.

> 7. Apple reviewer confuses a featureless blob of gray with CSAM material, several times A better collision won't be a grey blob, it'll take some photoshopped and downscaled picture of a kid and massage the least significant bits until it is a collision. https://openai.com/blog/adversarial-example-research/

So the person would have to accept and save an image that when looks enough like CSAM to confuse a reviewer…

Yes, the diligent review performed by the lowest-bidding subcontractor is an excellent defense against career-ending criminal accusations. Nothing can go wrong, this is fine.

Re: Hash collision in Apple NeuralHash model

#69

Post by cma »

sneak wrote: post #52
brigandish wrote: post #49

Earlier quoted context omitted.

I think you may have attracted less downvotes if the phrasing was changed to "Yes, just like false accusations of rape , it doesn't matter that you prove it was false afterwards." I also think that those downvoting you might've applied the principle of charity and taken the best interpretation of what you've written or at least ask .

All criminal accusations, including true ones , should be treated as false until the accused is proven guilty. This is a fundamental tenet of human rights in western, small-l liberal free societies. The fact that this is controversial these days is literally insane to me. The consequences of throwing this fundamental system out the window is that you get the sort of nonsense that happened with Assange, where he was l…

That's the burden for incarceration, not for making a personal best guess about guilt. Even far below best guess, would you send your kid with a camp councilor that you were 20% sure was guilty of something like that?

Re: Hash collision in Apple NeuralHash model

#70

Post by fsloth »

FabHK wrote: post #18

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6.…

"How can you use it for targeted attacks?"

Just insert a known CSAM image on target's device. Done.

I presume this could be used against a rival political party to ruin their reputation - insert bunch of CSAM images on their devices. "Party X is revealed as an abuse ring". This goes oh-so-very-nicely with Qanon conspiracy theories which even don't require any evidence to propagate widely.

Wait for Apple to find the images. When police investigation is opened, make it very public. Start a social media campaign at the same time.

It's enough to fabricate evidence only for a while - the public perception of the individual or the group will be perpetually altered, even though it would surface later that the CSAM material was inserted by hostile third party.

You have to think about what nation state entities that are now clients of Pegasus and so on could do with this. Not how safe the individual component is.

Post reply on HN