ImageNet contains naturally occurring Apple NeuralHash collisions • Hacker News Board

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#341

Post by mthoms » Thu, Aug 19, 2021, 9:08 PM UTC

foerbert wrote: post #258

mrtranscendence wrote: post #175

Earlier quoted context omitted.

> The difference is the ease with which they can demur. If Apple can be cowed by China into adding fake CSAM hashes by threat of banning iPhone sales, they could be cowed to surveil Chinese citizens in the search for subversive material. It's no skin off China's back if it's harder for Apple -- they'll either make the demand or they won't. This changes basically nothing.

I think that's a too simplistic view. It's kinda true, but ignores how humans really work. Apple will be pushed around to a degree, but there will be limits. The harder the ask now the less China can ask later. And the more Apple can protest about the difficulty and impossibility and other consequences they will face, the more likely China is to back off. Both sides want to have their cake and eat it too, and will co…

Apple does have some degree of leverage over the CCP too. I realize its not possible today... but in 3-5 years, Apple may be in a position to move some/all of their manufacturing elsewhere.

The direct job losses are one obvious problem for the CCP but a company like Apple saying "We're moving production to Taiwan/Vietnam/US because of security risks in China" would be catastrophic for the (tech) manufacturing industry as a whole in China. No sane Western based CEO will want to be seen taking that security gamble.

Do I think Apple would do that and forgo the massive Chinese smartphone market? That's another story.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#342

Post by Xamayon » Thu, Aug 19, 2021, 9:08 PM UTC

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

At this point, with all the easily producible collisions, the Gov't could just modify some CSAM images to match the hash of various leaked documents/etc they want to track. Then they don't even have to go thru special channels. Just submit the modified image for inclusion normally! (Not quite that simple, as they would still need to find out about the matches, but maybe that's where various NSA intercepts could help...)

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#343

Post by saithound » Thu, Aug 19, 2021, 9:08 PM UTC

ChrisKnott wrote: post #311

saithound wrote: post #299

Earlier quoted context omitted.

I suggest you reread the comment, because "people can send you images that are visually indistinguishable from known CSAM" is not what is being said at all. Where did you even get that from? The point is precisely that people can become victims of various new attacks, without ever touching photos that are actual "known CSAM". For Christ's sake, half the comments here are about how adversaries can create and spread po…

Can you explain how these theoretical political memes hash-match to an image in the NCMEC database, and then also pass the visual check? > "No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people." Did I say "taking"? I am talking about sending (theoretical) actual images from the NCMEC database. This is functionally id…

Yes, I can. This is just one possible strategy: there are many others, where different things are done, and where things are done in a different order.

You use the collider [1] and one of the many scaling attacks ([2] [3] [4], just the ones linked in this thread) to create an image that matches the hash of a reasonably fresh CSAM image currently circulating on the Internet, and resizes to some legal sexual or violent image. Note that knowing such a hash and having such an image are both perfectly legal. Moreover, since the resizing (the creation of the visual derivative) is done on the client, you can tailor your scaling attack to the specific resampling algorithm.

Eventually, someone will make a CyberTipline report about the actual CSAM image whose hash you used, and the image (being a genuine CSAM image) will make its way into the NCMEC hash database. You will even be able to tell precisely when this happens, since you have the client-side half of the PST database, and you can execute the NeuralHash algorithm.

You can start circulating the meme before or after this step. Repeat until you have circulated enough photos to make sure that many people in the targeted group have exceeded the threshold.

Note that the memes will trigger automated CSAM matches, and pass the Apple employee's visual inspection: due to the safety voucher system, Apple will not inspect the full-size images at all, and they will have no way of telling that the NeuralHash is a false positive.

[1] https://github.com/anishathalye/neural-hash-collider

[2] https://embracethered.com/blog/posts/2020/husky-ai-image-res...

[3] https://bdtechtalks.com/2020/08/03/machine-learning-adversar...

[4] https://graphicdesign.stackexchange.com/questions/106260/ima...

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#344

Post by int_19h » Thu, Aug 19, 2021, 9:08 PM UTC

ChemSpider wrote: post #303

Bjartr wrote: post #300

Earlier quoted context omitted.

> citizens are spied on by a country that is not them I thought countries often have under the table agreements with one another to explicitly spy on each others citizens, since its illegal for the country to spy on its own citizens. It's illegal for the other country too, but it's a lot easier to turn a blind eye to it.

The EU is not spying on any of its citizen. If you think otherwise, please link to some sources. Otherwise these are baseless rumors. These "everyone is doing it" statements are nonsense. Not everything is doing it.

Just one example:

https://en.wikipedia.org/wiki/Federal_Office_for_the_Protect...

https://www.spiegel.de/international/germany/infiltrating-th...

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#345

Post by wobbly_bush » Thu, Aug 19, 2021, 9:15 PM UTC

gsibble wrote: post #329

c7DJTLrn wrote: post #322

Earlier quoted context omitted.

I'm so done. I'm sorry to dump a pointless rant like this on HN but... what the hell is going on these days? Nobody seriously seems to care about legitimate privacy concerns anymore. If I were in a position of power, like being CEO, CTO, or even just an engineer on the team at Apple that implemented this, I'd do EVERYTHING to make sure that my power is in check and that I'm not pushing a fundamentally harmful technol…

The feeling is mutual. The devices we own are now being used to actively police us.

The parent comment might have misunderstood what the government asked for. It asked for a feature to "report spam" by end users.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#346

Post by int_19h » Thu, Aug 19, 2021, 9:19 PM UTC

rovr138 wrote: post #339

alfalfasprout wrote: post #2

There's been a lot of focus on the likelihood of collisions and whether someone could upload eg; an image with a matching hash to your device to "set you up", etc. But what's still extremely concerning is that there is still no guarantee that the hash list used can't be coopted for another purpose (eg; politically insensitive content).

On top of that, what happens if a court/government orders them to give them all the current data about people with matches, regardless of the 30 matches. They can't say if it is or not a match so they have to go after the individuals. Is that enough evidence for a warrant? Someone in the court thinks it's true and can't prosecute?, oh, it got leaked . -- Not every country has the same protections about innocent until…

More broadly speaking, every part of this scheme that is currently an arbitrary Apple decision (and not a technological limitation), can easily become an arbitrary government decision.

And yes, it's true that the governments could always mandate such scanning before. The difference is that it'll be much harder politically for Apple to push back against tweaks to the scheme (such as lowering the bar for manual review / notification of authorities) if they already have it rolled out successfully and publicly argued that it's acceptable in principle, as opposed to pushing back against any kind of scanning at all.

Once you establish that something is okay in principle, the specifics can be haggled over. I mean, just imagine this conversation in a Congressional hearing:

"So, you only report if there are 30+ CSAM images found by the scan. Does this mean that pedophiles with 20 CSAM images on their phones are not reported?"

"Well... yes."

"And how did you decide that 30 is the appropriate number? Why not 20, or 10? Do you maybe think that going after CSAM is not that important, after all?"

There's a very old joke along these lines that seems particularly appropriate here:

"Churchill: Madam, would you sleep with me for five million pounds?

Socialite: My goodness, Mr. Churchill… Well, I suppose… we would have to discuss terms, of course…

Churchill: Would you sleep with me for five pounds?

Socialite: Mr. Churchill, what kind of woman do you think I am?!

Churchill: Madam, we’ve already established that. Now we are haggling about the price."

Apple has put itself in the position where, from now on, they'll be haggling about the price - and they don't really have much leverage there.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#347

Post by Natsu » Thu, Aug 19, 2021, 9:19 PM UTC

Xamayon wrote: post #342

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

At this point, with all the easily producible collisions, the Gov't could just modify some CSAM images to match the hash of various leaked documents/etc they want to track. Then they don't even have to go thru special channels. Just submit the modified image for inclusion normally! (Not quite that simple, as they would still need to find out about the matches, but maybe that's where various NSA intercepts could help.…

Not quite, a CSAM hash match triggers another match within Apple to avoid false positives and then a human review. It wouldn't be trivial for them to extract matches out of that, and they'd only be able to track files they already know the contents for.

I would think they could more easily just make your phone carrier install a malware update on your phone, rather than jumping through all of these hoops to get them access they already have.

Plenty of data is leaking out of people's phones already as can be seen from, e.g. the Parler hack.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#348

Post by pharrington » Thu, Aug 19, 2021, 9:20 PM UTC

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Apple's CSAM detection has nothing to do with this. The vector for an authoritarian government getting blacklists into tech is that government telling the tech vendor "ban this content. We don't care how."

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#349

Post by mthoms » Thu, Aug 19, 2021, 9:20 PM UTC

rovr138 wrote: post #339

alfalfasprout wrote: post #2

There's been a lot of focus on the likelihood of collisions and whether someone could upload eg; an image with a matching hash to your device to "set you up", etc. But what's still extremely concerning is that there is still no guarantee that the hash list used can't be coopted for another purpose (eg; politically insensitive content).

On top of that, what happens if a court/government orders them to give them all the current data about people with matches, regardless of the 30 matches. They can't say if it is or not a match so they have to go after the individuals. Is that enough evidence for a warrant? Someone in the court thinks it's true and can't prosecute?, oh, it got leaked . -- Not every country has the same protections about innocent until…

As I understand it, Apple's servers know nothing until the 30+ match threshold is reached. This is actually one way that their system might be an improvement.

NB: I'm not in favour of this system - I'm only commenting on this one specific scenario.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#350

Post by nodamage » Thu, Aug 19, 2021, 9:20 PM UTC

Xamayon wrote: post #342

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

At this point, with all the easily producible collisions, the Gov't could just modify some CSAM images to match the hash of various leaked documents/etc they want to track. Then they don't even have to go thru special channels. Just submit the modified image for inclusion normally! (Not quite that simple, as they would still need to find out about the matches, but maybe that's where various NSA intercepts could help.…

All of the major tech companies already scan images uploaded to their services so isn't this already theoretically possible now? How is the situation changed by Apple using on-device scanning instead of cloud scanning (considering these images were going to be uploaded into iCloud anyway).