ImageNet contains naturally occurring Apple NeuralHash collisions • Hacker News Board

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#61

Post by tyingq » Thu, Aug 19, 2021, 5:33 PM UTC

orhmeh09 wrote: post #54

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Could you provide specific evidence that China has and would do this? I’ve a hard time recalling any specific cases. Maybe nation-states do this kind of thing, but I’m only aware of the countless times the United States has done this. What’s the recent history?

That China would use heavy handed tactics to coerce a US tech company?

Google for "Operation Aurora"

Also see: https://www.reuters.com/article/us-china-apple-icloud-insigh... for a very similar situation to the one described up-thread.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#62

Post by cmelbye » Thu, Aug 19, 2021, 5:33 PM UTC

slg wrote: post #52

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

I would expect Apple to say the same thing if the CCP proposed a system of scanning devices last month. I fail to see how this system changes the calculus for how Apple will deal with authoritarian governments. If Apple could stand up to them before this system, why can't they stand up to them with this system?

Apple doesn't stand up to authoritarian governments. They give direct access of their systems to the US and Chinese governments already.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#63

Post by programzeta » Thu, Aug 19, 2021, 5:33 PM UTC

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Apple’s trying to mitigate this by putting themselves to a more internationally focused standard of matches against at least two separate sources of CSA hashes, if my understanding of their announcements/follow-ups is right.

Separately, the US has even greater pressure on Apple in the case they want to unilaterally add database images, considering they have a actual chance and means to jail (and run through the legal ringer) whomever tells them ‘no’. And that’s just the overt pressure available; I think this is a more likely potential for trust violation here, even if both could come to pass.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#64

Post by dannyobrien » Thu, Aug 19, 2021, 5:35 PM UTC

endisneigh wrote: post #29

thedingwing wrote: post #20

Earlier quoted context omitted.

You don’t need to be sent to prison to be irreparably harmed by an accusation.

Ok sure, PhotoDNA has existed since 2008. Where are the instances of people being sent to prison?

One of the things that is happening now is that the entire PhotoDNA system is finally coming under the level of oversight that it should have had right from the start.

I can tell you from working in this area that it's possible for someone to have their lives ruined by a misplaced investigation, have that investigation abandoned because they turn out to be obviously innocent, and for that to not be well-known, because people simply would not understand the context.

Before this Apple scandal, if you'd written to your reepresentative or a journalist or an activist group and said "I was framed for child abuse because of computer program that misidentified innocent pictures", they would attach a very low priority to dealing with you or publicising this. And almost all people who have experienced this kind of nightmare really don't want to re-live it in public for some tiny possibility of real justice being served for them, or for others. They just want it to all go away.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#65

Post by danuker » Thu, Aug 19, 2021, 5:35 PM UTC

endisneigh wrote: post #43

biesnecker wrote: post #41

Earlier quoted context omitted.

They're not actively scanning your phone, they're actively scanning files you send them.

That’s the same as what Apple is going to do - scan right before sending to iCloud

The scanning on the device before the upload makes it easier to do surveillance.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#66

Post by skygazer » Thu, Aug 19, 2021, 5:35 PM UTC

jeffbee wrote: post #18

Following the topic because it's interesting, but I'm bothered by the chain of reasoning necessary to reach these conclusions. How sure are we that the means of dumping the model from iOS and converting it to ONNX results in a faithful representation of what NeuralHash actually does?

FWIW, Apple says that’s not even the NN hash function they’ll be using. This one has been buried in iOS since 2019, so it may have been a prototype. Although dumping the NN image seems straightforward, so this seems a faithful copy of that older hash function. (There’s been a suggestion that it’s sensitive to floating point processor implementations, so small hash differences may occur between different CPU architectures.)

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#67

Post by breuleux » Thu, Aug 19, 2021, 5:36 PM UTC

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Presumably Apple would be afraid that, say, the EU becomes suspicious, issues a court order to obtain the hashes, notices they cannot audit the CCP hashes, pointedly asks "what is this", becomes absolutely livid that their citizens are spied on by a country that is not them, fines Apple out the wazoo, then extradites whoever is responsible and puts them in prison. I mean, China's not the only player in this. Putting extra hashes to surveil Chinese citizens, yeah, they might do that, but it'd be suicide to put them anywhere else.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#68

Post by tyingq » Thu, Aug 19, 2021, 5:36 PM UTC

jeffbee wrote: post #18

Following the topic because it's interesting, but I'm bothered by the chain of reasoning necessary to reach these conclusions. How sure are we that the means of dumping the model from iOS and converting it to ONNX results in a faithful representation of what NeuralHash actually does?

I agree in theory, but the burden of proof shouldn't be on outsiders that have no other choice than to extrapolate. Proposing something like this running on an end-user's personally owned device should have a really high bar.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#69

Post by foerbert » Thu, Aug 19, 2021, 5:36 PM UTC

slg wrote: post #52

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

I would expect Apple to say the same thing if the CCP proposed a system of scanning devices last month. I fail to see how this system changes the calculus for how Apple will deal with authoritarian governments. If Apple could stand up to them before this system, why can't they stand up to them with this system?

The difference is the ease with which they can demur. Before, it would be a whole heck of a lot of new, additional work. They also have the problem of actually introducing it without being noticed, or having to come up with some cover for the new behavior.

Now? Well now it's real simple. It will even conveniently not expose the actual images it's checking for. Apple now has significantly less ability to rationally reject the request based on the effort and difficulty of the matter.

Even Apple's own reasons to reject the request have decreased. It would have legitimately cost them more to fulfill this request before, even if China did want to play hardball. Now, they have greater incentive to go along with it.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#70

Post by SilasX » Thu, Aug 19, 2021, 5:37 PM UTC

criticaltinker wrote: post #3

> In order to test things, I decided to search the publicly available ImageNet dataset for collisions between semantically different images. I generated NeuralHashes for all 1.43 million images and searched for organic collisions. By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2, I was able to compare the NeuralHashes of over 2…

>2 in 2 trillion image pairs

Reporting the collision rate per image pair feels misleading. What you really want to know is the number of false positives per image in the relevant set, not image pair, as that's the figure that indicates how frequently you'll hit a false positive.