Hash collision in Apple NeuralHash model

I can’t tell what would prevent a jailbroken device from pairing an illegal CSAM image with the safety voucher from a known-innocuous image, since the whole system depends on the trustworthiness of the safety voucher and that it truly represents the image in question. If the device is compromised I would think all bets would be off. To me, one potential flaw of this system may be that Apple inherently trusts the device. The existence of a jailbreak seems like a massive risk to the system.

In fact, Apple themselves generate fake/meaningless safety vouchers a certain percentage of the time (see synthetic safety vouchers.) If a jailbroken phone could trigger that code path for all images in the pipeline, apple’s system would be completely broken.

On the other hand, this may be just the excuse apple needs to lock down the phone further, to “protect the integrity of the CSAM detection system.” Perhaps they could persuade congress to make jailbreaking a federal crime. Perhaps they’re more clever than I ever imagined. Or perhaps they can fend off alternate App Store talk for sake of protecting the integrity of the system. Or perhaps staying up too late makes me excessively conspiratorial.

"Apple said that there is a one in one trillion chance of a false positive"

https://techcrunch.com/2021/08/05/apple-icloud-photos-scanni...

Versus:

https://blog.roboflow.com/nerualhash-collision/

Along with:

https://github.com/anishathalye/neural-hash-collider

Demonstrates blatant incompetence on the part of Apple's design. At this point, trust is broken. That's the end of credibility regarding any other claim made about the security or base function of the system.

As for a simple example of one of many potential attack vectors (search HN for "neuralhash" and read the comments for countless examples), here is a basic attack:

As Apple's hash algorithm is entirely broken and useless, a malicious actor can easily craft a NSFW, but otherwise legal image, that has a hash matching one in the CSAM database, that is visually ambiguous in that context (for example, a close-up). Targeted attacks could be launched on unsuspecting victims as simply as sending them an image.

If their app is configured to auto-save to iCloud - as some are by default, and as many are configurable to be - the image will then get automatically flagged as a match, and may well potentially pass a human review - because the image may indeed look just like CP - and an innocent person, who may not even be aware of the presence of the image in their iCloud library at this point, may get visited by the police.

That in itself is bad enough.

However it gets worse - do bear in mind that "the police" is not an equivalent concept across nations, yet iPhones are ubiquitously used across the world. In certain countries, a "visit from the police" over such automatically-flagged content could well result in presumption of guilt to the point of immediately administered punishment and a destroyed innocent life.

In China for example, conviction rates are routinely 99.99% - being accused of a crime there is equivalent to being convicted, and that's just one major country containing well over 100 million iphone users.

Apple is demonstrating not only gross incompetence in this design of this "system", but vagrant disregard for human rights, along with utterly destroying their own long-cultivated pro-privacy stance.

It's a disaster for them and their customers in every possible way it could be.

I don't see any mention of the reckless disregard standard in the New York v. Ferber decision, either? So far as I can see, it just says that CSAM is outside of the scope of 1A, so long as it's "adequately defined by the applicable state law". Am I missing something in the opinion?

I think I explained it adequately. An attacker gets some hashes likely to be in the database. The attacker modified legal pornography of young looking people, perhaps closeups of genitalia, to match the child porn hashes.

There are plenty of examples of US persons being charged for lawful pornography which a prosecutor accused of being child porn, I linked to one such example-- where the accused was rescued only by the testimony of the actress, and where the prosecution had expert witnessess testifying that the images of an adult were images of a child.

The attacker in this case would know the origin of the images and could point to them. The victim would have no idea where they came from.

I have searched the case law, and can find no case where the NCMEC provided an image for comparison. Are you aware of any?

You also seem to have continued to move the goalpost. Being merely accused of possessing child porn would be extremely damaging to a person. The fact that they might escape conviction by a jury after years of legal ordeal, incarceration, and having their reputation ruined, is not that much consolation.

> The idea that new images of nudity could be caught in it is because

This is an absurd claim. I have demonstrated (https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...) that it is possible to alter existing images to match an arbitrary neuralhash resulting in a perfectly normal looking image. This point is not a matter for dispute, anymore.

I could easily do so with a nude or pornographic image, I only chose to use SFW pictures out of good taste.

My interest in apple's scheme arouse out of their malicious use of cryptography to shield their actions from accountability. I didn't even hear about the imessage nudie scanner until some time after. You can be certain no such confusion applies here.

Imagine if our cognitive capability wasn’t exploited.