Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hash collision in Apple NeuralHash model

github.com

Post reply on HN
Print view View original

661–670 of 725 posts

Re: Hash collision in Apple NeuralHash model

#661

Post by cirrus3 »

FabHK wrote: post #18

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6.…

Are you being serious? #7 is literally "Apple reviewer confuses a featureless blob of gray with CSAM material, several times"

30 times.

30 times a human confused a blob with CSAM?

Re: Hash collision in Apple NeuralHash model

#662

Post by teachrdan »

cirrus3 wrote: post #660
Kalium wrote: post #194

Earlier quoted context omitted.

A police raid on a person's home, or even a gentler thorough search, can be enough to quite seriously disrupt a person's life. Certainly having the police walk away with all your electronics in evidence bags will complicate trying to work remotely. Of course, this is assuming everything works as intended and they don't find anything else they can use to charge you with something as they search your home. If you smoke…

The police do not show up until a human has compared the matched image to the actual. Just stop.

The more collisions, the more chances of a false positive by a (tired, underpaid) human. I don't envy the innocent person whose home gets raided by a SWAT team convinced they're busting a child sex trafficker.

Re: Hash collision in Apple NeuralHash model

#663

Post by meowster »

shapefrog wrote: post #658
icelancer wrote: post #650

Earlier quoted context omitted.

> I can guarantee nobody will see the inside of a courtroom This wasn't the question I asked.

> grey noise will never pass for CSAM > You sure that one or more human operators will never make this mistake. Yes I can say 100% that no human operator will ever classify a grey image for a child being raped. Happy to put money on it.

It's possible that the operator accidently clicks the wrong button.

When dealing with a monotonous task that the operator is probably getting PTSD from, I think the chance is greater than 0%.

Articles about content moderators and PTSD:

https://www.businessinsider.com/youtube-content-moderators-a...

https://www.bbc.com/news/technology-52642633

https://www.theverge.com/2019/2/25/18229714/cognizant-facebo...

Re: Hash collision in Apple NeuralHash model

#664

Post by brandon272 »

treis wrote: post #656
brandon272 wrote: post #653

Earlier quoted context omitted.

I'm not even sure where to start here. FedEx knows that CSAM is sent using its services in the same way that Apple does. That is to say that both companies know that CSAM has been sent historically and both know that is possible to send that type of material using its services, and one could argue that they should assume that their services are used to transfer that data. Why does one of these companies have to go to…

>If we suggest that Apple "knows" about CSAM on their network, we must also accept that Apple "knows" about many other crimes in which the devices they are sell are used in the planning and execution of those crimes. Why are they only focused on CSAM if they'd also have legal exposure in these other crimes simply for being a hardware and service provider? It's not knowledge of a crime. Apple is committing a crime by…

> Apple is committing a crime by possessing and distributing CSAM.

Apple's liability for content is limited by Section 230. If they find out about specific instances of CSAM on their servers they are required to remove it.

They are not required to hunt for it or otherwise search it out. They are not required to scan for it. Anything currently being done in that regard is voluntary.

> But that is exactly what they're doing. They are agreeing to possess and deliver images of which they know a portion are CSAM.

When you use Apple's cloud services, you agree to multiple legally binding agreements that include provisions about not using their cloud services to store or transmit illegal materials including CSAM.

Someone using Apple's services to do that in contravention of those agreements does not constitute them "agreeing to possess and deliver" that content.

What WOULD constitute them conspiracy and them agreeing to possess and deliver that content would be if they were informed of the specific instance of content being transmitted and delivered and did nothing about it or otherwise enabled its continued storage and distribution.

Re: Hash collision in Apple NeuralHash model

#665

Post by FabHK »

yayr wrote: post #644
cyanite wrote: post #639

Earlier quoted context omitted.

None of this makes the system useless or harmful. Also, it’s not Apple’s algorithm. The actual hash list Apple will use is not accessible to the device.

If anybody with enough motivation can modify any existing harmless image to have the same neural hash as a "tracked database" image this will create too many false positives. Too many false positives make the algorithm useless. If someone with even more motivation and the means to put those images onto your device via social engineering, exploits or maybe even features and you become the target of a criminal investig…

> this will create too many false positives. Too many false positives make the algorithm useless

Apple can (and will) run a second algorithm server side to filter out further false positives.

> If someone with even more motivation and the means to put those images onto your device via social engineering, exploits or maybe even features

Such an attacker could just plant CSAM directly. The hash collision has no bearing on it. If, however, it is hash collision you're worried about, they'd be caught during the manual review.

Re: Hash collision in Apple NeuralHash model

#666

Post by nullc »

hannasanarion wrote: post #651
nullc wrote: post #631

Earlier quoted context omitted.

I'm really surprised to see a fellow HN poster making such a stark failure to generalize. You see that they can do this with pictures of dogs. What makes you think they can't do exactly the same with pictures of crotches? Presumably you don't believe there is some kind inherent dog-nature that makes dog images more likely to undermine the hash. :) People are using pictures of dogs because they are a tasteful safe-for…

What kind of non-child-porn imagery is so similar to child porn that it fools the NCMEC investigator who is looking at it side-by-side with its maliciously-collisioned hash match, but is at the same time so so dissimilar from real child porn that, as you say, the target won't report its sudden appearance on their device and the the sender won't be at legal risk? Your scenario requires the image (not the hash, but the…

You're assuming that there is a side-by-side comparison with a hash matching image. I think that is an extremely big assumption which assumes facts not in evidence at all.

As far as what kind of image would be believed to be child porn without a side-by-side with the supposed match: ordinary porn. Without context plenty would be hard to distinguish, especially with the popularity of waxed smooth bodies and explicit close up shots.

Prosecution over "child porn" that isn't is already a proved thing, with instances where the government was happily trotting out 'experts' to claim that the images were clearly a child based on physical characteristics only to be rebuffed by the actual actress taking the stand. ( https://www.crimeandfederalism.com/page/61/ )

Re: Hash collision in Apple NeuralHash model

#667

Post by neolog »

floatingatoll wrote: post #655
neolog wrote: post #654

Earlier quoted context omitted.

I'd say the primary differences are that the CSAM scan is a perceptual hash rather than a regular file hash, and that the technical infrastructure of the CSAM system is designed from the ground up to be used against (rather than for) the user and report them individually to authorities for violation.

Do you have an alternate design in mind that is both "used for the user", and is also effective at reporting CSAM content being uploaded from the device, without allowing CSAM abusers to opt-out of that reporting? I haven't been able to come up with anything myself, but maybe you've had better luck.

I can only point to other people who know more than me.

https://stratechery.com/2021/apples-mistake/ is a smart tech commentator

https://www.nytimes.com/2021/08/11/opinion/apple-iphones-pri... are two security/encryption experts

Re: Hash collision in Apple NeuralHash model

#668

Post by brokenmachine »

cwizou wrote: post #265
tambourine_man wrote: post #209

Neuralhashes are far from my area of expertise, but I've been following Apple closely ever since its foundation and have probably watched every public video of Craig since the NeXT take over and here is my take: I've never seen him so off balance before as in his latest interview with Joanna Stern. Not even in the infamous “shaking mouse hand close up” of the early days. Whatever you say about Apple, they are an extr…

I think they clearly didn't anticipate that people would perceive it as anything but a breach of trust, that their device was working against them (even for a good cause, against the worst people). And because of this they calibrated their communication completely wrong, focusing on the on device part as being more private. Using the same line of thinking they use for putting Siri on device. And the follow up was an…

Thanks for the description.

That's a *huge* amount of crypto mumbo-jumbo for a system to scan your data on your own device and send it to the authorities.

They must really care about children!!

If only this system was in place while Trump, Jeffrey Epstein, and Prince Andrew were raping children, surely none of that would have happened!! /s

Re: Hash collision in Apple NeuralHash model

#669

Post by int_19h »

mannerheim wrote: post #89
nyuszika7h wrote: post #58

Earlier quoted context omitted.

The whole point flew over your head. If it's unchanged to the human eye then surely the human reviewer will see that it's a false positive?

No, it's important to point that out lest people think collisions can only be generated with contrived examples. I haven't studied neural hashes in particular, but for CNNs it's extremely trivial to come up with adversarial examples for arbitrary images. Anyway, as for human reviewers, depends on what the image being perturbed is. Computer repair employees have called the police on people who've had pictures of their…

Which would still be a privacy violation, since an actual human is looking at a photo you haven't consented to share with them.

Re: Hash collision in Apple NeuralHash model

#670

Post by int_19h »

hannasanarion wrote: post #550
falcolas wrote: post #484

Earlier quoted context omitted.

Here's the thing with CSAM - it's illegal to view and transmit. So nobody, until the police have confiscated your devices, will actually be able to verify that it is a "child being raped." They'll view visual hashes, look at descriptions, and so forth, but nobody from Apple will actually be looking at them, because then they are guilty of viewing and transmitting CSAM. I noted in another comment, even the prosecutors…

Where did you get this idea, scooby doo? It is not illegal to be an unwilling recipient of illegal material. If a package shows up at your door with a bomb, you're not gonna be thrown in jail for having a bomb.

Possession of CSAM is a strict liability crime in most jurisdictions.
Post reply on HN