Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

471–480 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#471

Post by XorNot »

mLuby wrote: post #404
incrudible wrote: post #400

Earlier quoted context omitted.

This leaves open the question of how the image gets on the device of the victim. You would have to craft a very specific image that the victim is likely to save, and the existence of such a specially crafted file would completely exonerate them.

I'd think the attack would be done in reverse: 1. Get a photo that the target already has. 2. Generate an objectionable image with the same hash as the target's photo. (This is obviously illegal.) 3. Submit the objectionable image to the government database. Now the target's photo will be flagged until manually reviewed. This doesn't sound impossible as a targeted attack, and if done on a handful of images that milli…

This requires the attacker handling CSAM which defeats the benefit. The risk in all cases is anytime you actually handle CSAM then the attack is void since you're now actually guilty of the crime and have to do it (very few will cross that line).

The point though is that this is something someone's Apple phone is doing, that their device is not. So the goal is to send a hash collided images by non-Apple channels (email) where there is a reasonably good chance that image would make it's way into someone's global device photo store and into automatic iCloud uploads.

Sending an MMS would work, for example, or a picture to Signal which then someone saves to outside of Signal (a meme).

In all these cases, the original sender doesn't have an Apple device: so they're not getting scanned by the same algorithm, but more importantly their device is not spying on them. Importantly too: they've done nothing illegal.

But: the victim is getting flagged by their own device. And the victim has to have their device seized and analysed to determine (1) that it's not CSAM, (2) that they were sent those images that flagged and aren't trying to divert attention by getting themselves false pinged upfront, but then (3) the sender has committed no crime. There's no reason or even risk to investigate them, because by the time the victim has dealt with law enforcement, it's been established that no one had anything illegal.

It's the digital equivalent of a sock of cat litter testing positive as being methamphetamine, except if it was your drive through McDonald's order.

The goal is not to get convictions, the goal is harrassment.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#472

Post by zionic »

mrtranscendence wrote: post #179
saithound wrote: post #163

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino…

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

1) be a horrible human and want to troll

2) modify close up/ambiguous adult porn to be flagged as CP with free GitHub tool

3) batch a few thousand porn photos like this to poison them

4) upload them everywhere, 4chan/Reddit/tumblr/discord/imagefap

5) some poor sap manages to save 20+ of your bait images

6) apple reviewer sees 100x100px blurry gray image of definitely porn that was flagged as CP. hits report.

7) a SWAT team bust down your door and takes all your devices while you go to jail and your mugshot is everywhere

Someone will do this, and the poisoned images will spread organically until they find a victim

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#473

Post by scraptor »

throwaway672000 wrote: post #291
mrtranscendence wrote: post #251

Earlier quoted context omitted.

Fair enough. I suppose it's true that you could create a colliding sexually explicit image where age is indeterminate, and the reviewer may not realize it isn't a match. > Given the ability to produce hash collisions, an adversary can easily generate photos that fail this visual inspection as well. Apple could easily fix this by also showing a low-res version of the CSAM image that was collided with, but I'll grant t…

Won't enough images be real matches for them to be looking at it (in low res) for most of their work day?

How stupid do you have to be to run your actual child porn operation on icloud after a week of headlines like this?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#474

Post by hda2 »

nonbirithm wrote: post #462
short_sells_poo wrote: post #327

Earlier quoted context omitted.

So in your mind because bad thing X is already happening, it's completely OK for bad thing XY to also start happening?

No, it's that in spite of there already being an invasive scanning process in place for this long at every major tech company that handles user data, nobody seemed to care until now.

Most people weren't aware or couldn't see where this was going. Now that it's becoming obvious, people are starting to care.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#475

Post by mLuby »

XorNot wrote: post #471
mLuby wrote: post #404

Earlier quoted context omitted.

I'd think the attack would be done in reverse: 1. Get a photo that the target already has. 2. Generate an objectionable image with the same hash as the target's photo. (This is obviously illegal.) 3. Submit the objectionable image to the government database. Now the target's photo will be flagged until manually reviewed. This doesn't sound impossible as a targeted attack, and if done on a handful of images that milli…

This requires the attacker handling CSAM which defeats the benefit. The risk in all cases is anytime you actually handle CSAM then the attack is void since you're now actually guilty of the crime and have to do it (very few will cross that line). The point though is that this is something someone's Apple phone is doing, that their device is not. So the goal is to send a hash collided images by non-Apple channels (ema…

> the sender has committed no crime

> they've done nothing illegal

Perhaps that's true in the narrowest sense, but aren't the odds of generating a colliding file so low as to all but rule out coincidence and therefore strongly indicate premeditated cyber-attack (which is illegal)?

If I were law enforcement, at the very least I'd want to keep tabs on these sources of false positives. Probably easy enough to convince a judge that someone capable of the "tech wizardry" to collide a hash can un-collide one too, and therefore more thorough/invasive search warrants of the source are justified.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#476

Post by j16sdiz »

0x0 wrote: post #391
mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

Seeing how "well" app review works, I would not be surprised if the "peon" sometimes clicks the wrong button while reviewing, bringing down a world of hurt on some innocent apple user, all triggered by the embedded snitchware running on their local device.

I guess there are thousands of App reviews each week, each one take tens of minutes to test..

The CASM thing, I guess, give at most a dozen a week...

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#477

Post by hda2 »

endisneigh wrote: post #30
Unklejoe wrote: post #27

Earlier quoted context omitted.

If we reduce it down to "someone being accused of a crime but later being found innocent", then there are several, and when it comes to sex crimes, the accusation alone is enough to ruin someone. People don't care about minor details such as the fact that the person was later found innocent. To them, it's the same as getting off on a technicality.

Ok yes - now then, PhotoDNA has existed since 2008. When has what you’ve described ever happened?

PhotoDNA relies on perceptual hashing?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#478

Post by labcomputer »

mrtranscendence wrote: post #192
zbobet2012 wrote: post #16

Earlier quoted context omitted.

That's not the route in which this will be exploited, and used at scale. A corrupt government has to know they want you "to just get you". Instead they will embed a collision in anti-government meme. That collision will flag you, and now they know you harbor doubts and will come get you. This is why it's a privacy concern. It's no the tech (like you said photo dna's been about forver), it's the scanning of the phone.

> That collision will flag you, and now they know you harbor doubts and will come get you. Only if that government has worked out some deal with Apple whereby such an anti-government meme would end in the government being notified accordingly. Don't forget that you need a sufficiently high number of collisions, for one, and that those collisions are audited by Apple before being sent to law enforcement.

Why do they need to work out a deal with Apple? Just bribe or blackmail the minimum wage reviewers.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#479

Post by iszomer »

wbl wrote: post #443
Notanothertoo wrote: post #325

Earlier quoted context omitted.

This, these charges are damning once they are made. Plus the countless legal dollars you are going to have to front and hours spent proving innocence and that's assuming the justice system actually works.. Try explain this to your employer while you start missing deadlines due to court dates.. The police also could easily leverage this to warrant hop. As they have been found doing in the past. I think the bike rider…

A false report is not going to lead to charges. In order to bring a case they need evidence of possession.

Innocent until proven guilty or, with the way we're heading, guilty until proven innocent. And it wouldn't matter as you are now already _tainted_.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#480

Post by theluketaylor »

heavyset_go wrote: post #448
theluketaylor wrote: post #229

Earlier quoted context omitted.

There are a lot of really valid criticisms of Apple plan here, but Apple has gone out of their way to prevent that exact case. Apple is using secret splitting to make sure they cannot decode the CSAM ticket until the threshold is reached. Devices also produce some synthetic matches to prevent themselves Apple (or anyone else) inferring a pre-threshold count based on the number of vouchers. https://www.apple.com/child…

> There are a lot of really valid criticisms of Apple plan here, but Apple has gone out of their way to prevent that exact case. Not they haven't. The files are still uploaded to Apple's servers where Apple holds their encryption keys. Apple or the FBI could scan the photos for CSAM whenever they want to.

The contortions apple is going through to scan locally only really make sense if they are gearing up to do end to end encryption of iCloud photos. I imagine they have other prerequisites before they are ready to announce that as a feature, but if they are not planning to encrypt more why wouldn’t they just scan in the cloud?
Post reply on HN