Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

521–530 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#521

Post by BizarroLand »

giantrobot wrote: post #254
mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a legitimate CSAM image and modify it such that when scaled for use in the perceptual algorithm(s) it changes to be some politically sensitive image. Upon review it'll look like CSAM so off it goes to reporting agencies. Because the perceptual hash algorithms are…

If you know the hash of an approved image, say of a puppy, a malicious person could edit a CP image to have the same hash as the puppy, right?

Wouldn't that mean that the system can be poisoned? How would they deal with that?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#522

Post by BizarroLand »

incompatible wrote: post #453
incrudible wrote: post #395

Earlier quoted context omitted.

> You've cost Apple time and money by making them review each such fake report. Ok, but… how do I profit? If I wanted to waste Apple employee time, I could surely find a way to do it, but why would I? The functioning of society relies on the fact that people generally have better things to do than waste each others time.

Perhaps somebody could make unflaggable CSAM images by giving them the same hashes as very widely available images.

Or you could identify the factors that cause a hash to be computed and then start generating random images that compute to the same hash, creating 10s of thousands of images of digital noise that all look alike to the computer.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#523

Post by ec109685 »

saithound wrote: post #506
ec109685 wrote: post #492

Earlier quoted context omitted.

The first half of your post is predicated on it being likely the noise added to generate hash A using the NeuralHash is likely to produce a specific hash B with some unknown perceptual hashing function (which they specifically call out [1] as independent of the NeuralHash function precisely because they don’t want to make this easy, so speculating it might be the NeuralHash run again is incorrect). Hash A is generate…

> Until you can show an example or paper where the same adversarial image generates a specific hash value for two unrelated perceptual hash functions, with one being hidden, it is not right to predict a high likelihood of that first scenario being possible. "There is no paper attacking ROT13 done twice, therefore it must be secure". Usually, it's on the one proposing the protocol to make a case for its security. Doub…

They claim their H is resistant to adversarial attacks, so they are claiming this to be difficult.

If I took an exact public perceptual hash function implementation and used that as H in your contest, it might be possible for a researcher attacking all public perceptual hash functions to stumble on the right one within 40 days.

I agree with you that we are trusting Apple to implement this competently. This isn’t something that can be proved to work mathematically where nothing about the implementation has to be kept secret.

So worse case everything you say could come true but to imply that is likely is wrong.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#524

Post by giantrobot »

BizarroLand wrote: post #521
giantrobot wrote: post #254

Earlier quoted context omitted.

It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a legitimate CSAM image and modify it such that when scaled for use in the perceptual algorithm(s) it changes to be some politically sensitive image. Upon review it'll look like CSAM so off it goes to reporting agencies. Because the perceptual hash algorithms are…

If you know the hash of an approved image, say of a puppy, a malicious person could edit a CP image to have the same hash as the puppy, right? Wouldn't that mean that the system can be poisoned? How would they deal with that?

Just a correction for you, there's not a list of approved images. The CSAM database are a list of illegal (unapproved if you will) images.

Other than that, yes it's possible to add noise to an image so a perceptual algorithm misidentifies it. I described the false positive case but it can also be used for false negatives. Someone can apply noise to a legit CSAM image (in the NCMEC database) so Apple's system fails to identify it.

The false positive case is scary because if it happens to you your life is ruined. The false negative case just means people have CSAM and don't get found by the system. I'm much more concerned about the false positive case.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#525

Post by Threeve303 »

I’m honestly fine with Apple doing this if it’s for the stated reasons. But experience has shown me that this is essentially copyright infringement enforcement dressed up in some emotional blackmail. Of course the point where this became inevitable was when Apple began creating and distributing media.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#526

Post by saithound »

giantrobot wrote: post #524
BizarroLand wrote: post #521

Earlier quoted context omitted.

If you know the hash of an approved image, say of a puppy, a malicious person could edit a CP image to have the same hash as the puppy, right? Wouldn't that mean that the system can be poisoned? How would they deal with that?

Just a correction for you, there's not a list of approved images. The CSAM database are a list of illegal (unapproved if you will) images. Other than that, yes it's possible to add noise to an image so a perceptual algorithm misidentifies it. I described the false positive case but it can also be used for false negatives. Someone can apply noise to a legit CSAM image (in the NCMEC database) so Apple's system fails to…

Keep in mind that there are multiple straight paths from the false negative case to the false positive case. I'll give you one examples: pedos can and will use the collider to produce large batches of CSAM that collide with perfectly legitimate images (e.g. common iPhone wallpapers). They literally have nothing to lose by doing this.

Eventually, these photos will make their way into the NCMEC database, and produce a large number of false positives. This will also make the other attacks discussed here easier to execute (e.g. by lowering the human review threshold, since everybody will start with a few strikes).

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#527

Post by FranksTV »

sennight wrote: post #444
skygazer wrote: post #420

Earlier quoted context omitted.

This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it. Granted, Apple makes no mention of any safeg…

The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after…

Why not just put actual child porn on their phone if you have access and want to screw with them?

And if you can do that, why wait for apple to implement this? Why not just put child porn on their phone and then call the police?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#528

Post by cinquemb »

FranksTV wrote: post #527
sennight wrote: post #444

Earlier quoted context omitted.

The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after…

Why not just put actual child porn on their phone if you have access and want to screw with them? And if you can do that, why wait for apple to implement this? Why not just put child porn on their phone and then call the police?

> Why not just put child porn on their phone and then call the police?

That does not scale.

> … why wait for apple to implement this?

Enables scaling of what the above commentator discusses.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#529

Post by cheald »

syshum wrote: post #446
cheald wrote: post #384

Earlier quoted context omitted.

Why would anyone bother calling the cops and telling them that someone they don't like is an imminent threat? The end result is that some officer just has to stop by and see that they aren't actually building bombs. You've cost someone a bit of time, but that's it.

Reality shows that not to be the case. How many "SWATTING" incidents now has resulted in much more than "some officer stopping by"....

My point precisely.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#530

Post by GatorD42 »

judge2020 wrote: post #439
GatorD42 wrote: post #437

Wouldn’t the most likely attack be an angry (ex) spouse with physical access to a target’s phone placing images on it (and uploading to iCloud)? I haven’t seen this seemingly likely scenario discussed much.

two scenarios: - Those images would either have to be actual CSAM (illegal to distribute in the first place and generally hard to get) - fake collision images are placed instead, and all that would do is push the multiple fake CSAM images to Apple's human reviewers who would then likely mark it as a false positive/spam. If the Apple reviewer makes a mistake and does report it as CSAM them the only risk (in the US) is…

The first scenario seems much more plausible and genuinely worrying.
Post reply on HN