Live data from Hacker News

A catalog of naturally occurring images whose Apple NeuralHash is identical

github.com

21–30 of 304 posts

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#21

I’m glad that people are trying to figure out any technical flaws in the system as best they can, but if I’m being honest I do trust Apple’s engineers to have built something that is solid from a technical stand point. Am I correct in that the primary reason folks are so upset is that the system could (probably) be easily modified such that -any- content could invoke legal action? That the main problem is really the…

Governments don’t get to search your house because some people out there have CP at home. Why should your smartphone be different?

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#23
post #10

Earlier quoted context omitted.

You (or at least Apple's customers) trust in and rely on Apple's proprietary software to do its job all the time. How is this different? I find this argument very weak.

1. You can at least somewhat audit the software running on an iPhone, for example by means of reverse engineering. You can’t audit the server side. 2. It’s one thing to rely on proprietary services like Find My or Siri. It’s another thing to rely on a secret server-side app that has the power to destroy your life.

"Hey Siri, accuse me of something that will ruin my life and reputation even if I'm not guilty".

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#24
post #12
post #9

Earlier quoted context omitted.

The point in the NeuralHash and PSI system is to preserve user privacy as far as possible. From a technical standpoint, it is not essential - a NeuralHash function that returns 0x0000… for everything would still catch CSAM. It's just that it would upload every single image on the user's device. Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the ori…

It doesn't really matter whether all images are uploaded, or just 1 in x (for large value of x), due to the Panopticon effect.

Let's not forget what the alternative is: this is about images that are uploaded on icloud anyway. The alternative is to upload the image in clear (or with ane encryption key that apple controls), and let apple run the CSAM filter on their servers.

Apple now has the ability to encrypt the images before sending them to icloud, with a private key you own. Except that some percentage of images that match the CSAM fingerprint with their neural feature extractor will be sent to a CSAM filter on the server side (whose workings we don't have many details about)

This whole thing backfired on Apple entirely due to psychological effects, not because they are really doing anything more "panopticon" that they would already able to do now on their icloud storage (after all people are ready sending their photos to apple)

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#25
post #3

Sigh, for the last time, it doesn't actually matter if the NeuralHash is identical. You need multiple images matching, and then the images are compared by another system on Apple's end, which you don't know anything about. The system is specifically designed so that colliding images does not pose a threat to the user. NeuralHash and the CSAM scanning is grotesque, but please, criticize it for what it is, not some bul…

Discussing the preimage attack on NeuralHash is not technical ignorance. Dismissing the preimage attack as irrelevant is.

0. Most importantly: the existence of a preimage attack makes Apple's system completely useless for its original purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. Thanks to the preimage attack, it's now completely useless for its nominal function! Apple put a lot of effort into a system that reduced the privacy and security of all their customers, and made the company itself more exposed to the whims of governments. And for no gain whatsoever.

1. There are no known perceptual hash functions on which preimage attacks are difficult. Barring a major "secret cryptographic breakthrough", Apple's second hash function is not resistant to preimage attacks either. In fact, the second algorithm is almost certainly easier to attack than NeuralHash itself, since it has to work on the "visual derivative", a fixed-size low-resolution thumbnail of the original image.

2. But isn't Apple's second algorithm kept secret, making it difficult to perform preimage attacks against it? No.

First of all, the second algorithm cannot be kept secret. Apple doesn't have its own CSAM database (the whole point is that they don't want to deal with CSAM on their servers!), so the algorithm has to be shared with multiple organizations which do have such databases, so that they can pre-compute the hashes that Apple will match against. Due to Apple's policy, some of these organizations will be located outside the US [1]. Chances are, the hash function will leak: Apple won't know if and when that happens.

Secondly, this _is_ security by obscurity. Some people argue that keeping the hash algorithm secret is similar to keeping a cryptographic key secret. This is not the case. Of course, any security system relies on keeping _something_ secret, but these secret somethings are not created equal. The secret keys of cryptographic algorithms are designed to satisfy Kerckhoffs's assumption. This means that the key, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system, even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs).

The second hash does not have this property at all. Keeping the algorithm secret does not ensure the confidentiality or integrity of Apple's system. E.g. if somebody gets access to a reasonable number of inputs-output examples, that allows them to train their own model which behaves similarly enough to let them find perceptual hash collisions, even if they don't know the exact details of the original algorithm. This is incredibly hard for cryptographic hashes, but very easy for perceptual hashes, since a small change in the input should cause only a small change in the output of the perceptual hash algorithm. So, to maintain security, Apple doesn't have to keep just the hash algorithm (or its configuration parameters) secret, but all the inputs and outputs as well. This is bad: the fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security.

Finally, the second hash algorithm is unlikely to be original (NeuralHash was original, and by all accounts it was a massive effort). If an attacker successfully guesses that Apple's secret algorithm H is closely related to a known algorithm, say PhotoDNA, they will probably be able to make a transfer attack against it. By engineering a PhotoDNA collision on the resized thumbnail (e.g. via a resizing attack, extensively discussed in a previous thread [3]), they have a reasonable chance of generating a H-collision as well. How good is fairly good? Well, something like 5% is more than enough! The attacker needs to produce a certain number of NeuralHash collisions (say 30 images) to get through the first threshold of Apple's algorithm. But after that, Apple will decode all the thumbnails in the user's safety voucher: the attacker only needs one of those 30 to get through the second hash. Given a sufficiently high probability of hash collisions, this can be achieved "blindly".

3. It's incredibly easy to come up with these kinds of attacks. Even the HN audience could come up with several reasonable plans, and could point out several reasonable issues, in two weeks. People who do malice for a living will have a much easier time with it. Even if somehow all the plans presented on HN turned out to be unviable, it will not take long for someone to stumble upon something practical. Any reassurance that Apple could provide at this point is fake. Cf. the timelines for real security: it took 17 years to come up with an analogous attack against SHA-1 [4], and two years after that to turn it into something that can be exploited in practice [5]. The existence of a preimage attack made Apple's system completely useless for its original purpose in two weeks. It's now just a security and privacy hole, with no other function. Keeping it around would be a travesty, even if it was difficult to exploit. But it's not.

[1] https://www.itnews.com.au/news/apple-to-only-seek-abuse-imag...

[2] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle

[3] https://news.ycombinator.com/item?id=28236102

[4] https://security.googleblog.com/2017/02/announcing-first-sha...

[5] https://www.zdnet.com/article/sha-1-collision-attacks-are-no...

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#26
post #9

Earlier quoted context omitted.

The point in the NeuralHash and PSI system is to preserve user privacy as far as possible. From a technical standpoint, it is not essential - a NeuralHash function that returns 0x0000… for everything would still catch CSAM. It's just that it would upload every single image on the user's device. Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the ori…

The “actual argument” against the system that this provides is that Apple lied about the likelihood of hash collisions. Therefore, why trust any of their other claims?

Apple's claims are based on the statistical likelihood that there would be 30 collisions with CSAM hashes within one user account.

Just because someone has found an image of a nearly featureless diagonal thing which collides with another image of a nearly featureless diagonal thing, that doesn't disprove Apple's claims.

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#27

I’m glad that people are trying to figure out any technical flaws in the system as best they can, but if I’m being honest I do trust Apple’s engineers to have built something that is solid from a technical stand point. Am I correct in that the primary reason folks are so upset is that the system could (probably) be easily modified such that -any- content could invoke legal action? That the main problem is really the…

Governments don’t get to search your house because some people out there have CP at home. Why should your smartphone be different?

This sums up the frustration very eloquently.

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#28
"This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2)"

That is not bad. As a tool to filter down what apple human reviewers need to look at this is pretty good.

Ultimately these images will make it to a human reviewer who can make a call as they would in any flagging system.

Could a backend server side system do a more precise hash (96 bits is not a ton) prior to human review?

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#29
post #10

Earlier quoted context omitted.

You (or at least Apple's customers) trust in and rely on Apple's proprietary software to do its job all the time. How is this different? I find this argument very weak.

1. You can at least somewhat audit the software running on an iPhone, for example by means of reverse engineering. You can’t audit the server side. 2. It’s one thing to rely on proprietary services like Find My or Siri. It’s another thing to rely on a secret server-side app that has the power to destroy your life.

What I somehow fail to grasp in the first argument is that this whole system is designed specifically so that it runs client-side. AFAIK all the alternatives (as in « cloud photo services ») have been doing the exact same thing on the server side for decades. If you upload your photos to the cloud, a lot of service actually already have the power to destroy your life.

Re: A catalog of naturally occurring images whose Apple NeuralHash is identical

#30
post #9

Earlier quoted context omitted.

The point in the NeuralHash and PSI system is to preserve user privacy as far as possible. From a technical standpoint, it is not essential - a NeuralHash function that returns 0x0000… for everything would still catch CSAM. It's just that it would upload every single image on the user's device. Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the ori…

The “actual argument” against the system that this provides is that Apple lied about the likelihood of hash collisions. Therefore, why trust any of their other claims?

I find this an unconvincing argument as well, you're saying that because Apple made a false claim, any claim may be valid. This is obviously not the case, what they did was to, albeit likely knowingly, calculate the hash collision probability /if each bit is a coin flip/, which comes out to pow(2, -k) for k bits. It's tiny. Of course, each bit is /not/ an independent coin flip under the NeuralHash function.

So again the actual argument becomes: what is that distribution like?

Post reply on HN