Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

491–500 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#491

Post by hypertele-Xii »

mrtranscendence wrote: post #179
saithound wrote: post #163

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino…

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

Imagine costing everyone a bit of privacy all the time.

Does that sound consistent with how Apple has positioned itself in the market?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#492

Post by ec109685 »

saithound wrote: post #489
ec109685 wrote: post #468

Earlier quoted context omitted.

This attack doesn’t work. If the resized image doesn’t match the CSAM image your NeuralHash mimicked, then when Apple runs it’s private perceptual hash, the hash value won’t match the expected value and it will be ignored without any human looking at it.

We have no reason to believe that Apple's second, secret perceptual hash provides any meaningful protection against such attacks. At best, we can hope that it'll allow early detection of attacks in a few cases, but chances are that's the best it can do. We might not ever learn: Apple now has a very strong incentive not to admit to any evidence of abuse or to any faults in their algorithm. (Sorry, this is going to be…

The first half of your post is predicated on it being likely the noise added to generate hash A using the NeuralHash is likely to produce a specific hash B with some unknown perceptual hashing function (which they specifically call out [1] as independent of the NeuralHash function precisely because they don’t want to make this easy, so speculating it might be the NeuralHash run again is incorrect). Hash A is generated via thousands of iterations of an optimization function, guessing and checking to produce a 12 bit number. What shows that same noise would produce an identical match when run through a completely different hashing function that is designed very differently specifically to avoid these attacks? Just one bit of difference will prevent a match. Nothing you’ve linked to would show any likelihood of that being anywhere close to 10 percent.

For the second part, yes if an Apple engineer (that had access to this code) leaked the internal hash function they used or a bunch of example image’s to hash values, that would allow these adversarial attacks.

Until you can show an example or paper where the same adversarial image generates a specific hash value for two unrelated perceptual hash functions, with one being hidden, it is not right to predict a high likelihood of that first scenario being possible.

Here’s a thought exercise, how long would it have taken researches to generate a hash collision with that dog image if the NeuralHash wasn’t public and you received no immediate feedback that you were “right” or getting closer along the way?

[1] https://www.apple.com/child-safety/pdf/Security_Threat_Model...

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#493

Post by bildung »

mrtranscendence wrote: post #179
saithound wrote: post #163

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino…

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

Because people behind the keyboards make mistakes all the time. Just in the last month i experienced

* a call center agent at a haulage firm, instead of entering the delivery date we talked about on the phone, clicked for the delivery to be returned to the factory.

* Google automatically blocked an ad account from delivering ads because we allegedly profiteered from Covid (untrue of course, but we surly talked about the challenges caused by the pandemic somewhere on the site, so the "AI" apparently got triggered by some keywords), and humans repeatedly confirmed the AI decision.

* Facebook blocked an ad account that was unused in 2020, wanted ID, got the correct ID (identical name etc.), and the human denied confirmation.

Google and Facebook are of course known to be beyond kafkaesque, so this is no surprise. But imagine the costs the innocents pay once they accidently get entered into the FBI CP suspect database.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#494

Post by rapnie »

zionic wrote: post #472
mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

1) be a horrible human and want to troll 2) modify close up/ambiguous adult porn to be flagged as CP with free GitHub tool 3) batch a few thousand porn photos like this to poison them 4) upload them everywhere, 4chan/Reddit/tumblr/discord/imagefap 5) some poor sap manages to save 20+ of your bait images 6) apple reviewer sees 100x100px blurry gray image of definitely porn that was flagged as CP. hits report. 7) a SWA…

Plus by doing this at scale you generate such workload for reviewers that they are way more likely to quickly press 'report' in step 6.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#495

Post by saithound »

Spooky23 wrote: post #418
saithound wrote: post #299

Earlier quoted context omitted.

I suggest you reread the comment, because "people can send you images that are visually indistinguishable from known CSAM" is not what is being said at all. Where did you even get that from? The point is precisely that people can become victims of various new attacks, without ever touching photos that are actual "known CSAM". For Christ's sake, half the comments here are about how adversaries can create and spread po…

> No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people. An attacker can embed a matching image inside of a PowerPoint zip file, and email it to any corporate employee using O365. Or, an angry parent can call the police and let them know that a 16 year old possesses nose pictures of their 15 year old girlfriend. The o…

Sure, your proposed attack, that requires the victim to have a 15 year old girlfriend, to break an (admittedly silly) law by having nude photos on their phone, for you to call the cops, and for them to take such a call seriously is clearly comparable to a vector that can be used to target innocents, groups of individuals, etc. who did not break the law in any way, and that do not require the attacker to handle prohibitex material at all, and requires Apple to keep a ton of information completely obscure to even provide a weak semblance of security (it was shown to be completely broken except possibly for one unknown hash, in two weeks). Clearly comparable. Sure. Clearly.

For one last time, the NeuralHash collisions make this tool perfectly unusable for catching pedos: all of the next generation of CSAM content will collide with hashes of popular, innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. It's completely useless for its nominal function. This would be a massive, hilarious own goal from Apple even if the public reaction was over the top (although it isn't). They just reduced the privacy and security of nearly all their customers, further exposed themselves to the whims of governments, and for no gain whatsoever.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#496

Post by perryizgr8 »

incrudible wrote: post #395
csmpltn wrote: post #242

Earlier quoted context omitted.

> "The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it." This can be abused to spam Apple's manual review process, grinding it down to a halt. You've cost Apple time and money by making them review each such fake report.

> You've cost Apple time and money by making them review each such fake report. Ok, but… how do I profit? If I wanted to waste Apple employee time, I could surely find a way to do it, but why would I? The functioning of society relies on the fact that people generally have better things to do than waste each others time.

> but why would I?

For the lulz.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#497

Post by GistNoesis »

ec109685 wrote: post #466
csmpltn wrote: post #242

Earlier quoted context omitted.

> "The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it." This can be abused to spam Apple's manual review process, grinding it down to a halt. You've cost Apple time and money by making them review each such fake report.

It can’t be. There’s a different private hash function that also has to match that particular csam image’s hash value before a human sees it. An adversarial attack can’t produce that one since the expected value isn’t known.

This second "secret" hash function, because it is applied to raw offensive content that Apple can't have, has to be shared at least with people maintaining the CSAM database.

You can't rely that it won't ever leak, and when it does, it will be almost undetectable and have huge consequences.

As soon as the first on-device CSAM flag has been raised, it becomes a legal and political problem. Even without a second matching hash, it already put Apple in an untenable position. They already are in a mud fight with the pigs.

They can't say : we got 100M hits this month on our first CSAM filter but we only reported 10 cases, because to avoid false positives our second filter throw everything to dev/null, and we didn't even manually reviewed them because your privacy matter to us. It has become a political problem where for good measure they will have to report cases to make the numbers look "good".

Attackers of the system can also plant false negatives aka real CSAM that has been modified enough to pass the first hash but fail this second hash. So that, in the audit, independent security researchers who review Apple system, will be able to say that Apple automated system, sided with the bad guys, by rejecting true CSAM and not reporting it.

Also remember, that Apple can also do something else than what they say they do for PR reasons : maybe some secret law will force them to reveal to the authorities as soon as the first flag has been raised, and force them not tell about it. And because it's in the name of fighting the "bad guys", that's something most people expect them to do.

From the user perspective, there is nothing we can audit, it's all security by obscurity disguised with pseudo-crypto-PR, it's just a big "Trust us" blanked signed paper that will soon be used to dragnet surveil anyone for any content.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#498

Post by raxxorrax »

saithound wrote: post #163

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino…

This also server as a pretext to deeply search someone's device. So you must expect your device getting randomly searched by law enforcement. Completely ridiculous.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#499

Post by WesolyKubeczek »

nonbirithm wrote: post #462
short_sells_poo wrote: post #327

Earlier quoted context omitted.

So in your mind because bad thing X is already happening, it's completely OK for bad thing XY to also start happening?

No, it's that in spite of there already being an invasive scanning process in place for this long at every major tech company that handles user data, nobody seemed to care until now.

Also, there's a difference. You upload stuff to someone's server' if they don't vet it, they become accomplices. Apple intrudes on what you do on your phone, this, among other things, tells that you paid mad bucks for this phone and you don't even own it, you rent it from your phonelord Apple. Also, in their eyes you're suspect and likely a filthy pedo.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#500

Post by raxxorrax »

KallDrexx wrote: post #356
biesnecker wrote: post #41

Earlier quoted context omitted.

They're not actively scanning your phone, they're actively scanning files you send them.

That's not actually answering the question in the GP about why this is different. Photos people send me to my Android are automatically sent through 3rd parties, either through MMS, Facebook messenger, Google Photos, or One Drive. Photos arriving on my device are almost guaranteed to be uploaded to both OneDrive and Google Photos based on how defaults of Android phones are setup. So someone could already send hash co…

Why waste a resource like that and throw them in prison when you have compromising material? That would be stupid. Epstein was probably a high society pimp and there probably was enough evidence for convictions. That wouldn't have happened if it didn't get public.
Post reply on HN