Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hash collision in Apple NeuralHash model

github.com

Post reply on HN
Print view View original

331–340 of 725 posts

Re: Hash collision in Apple NeuralHash model

#331

Post by robertoandred »

short_sells_poo wrote: post #311
stetrain wrote: post #224

Earlier quoted context omitted.

This system does not use ML to find new CSAM images. It only checks for ones already in a known database. Your pictures of kids in the bathtub are not on the list. What is show to the reviewer is a "visual derivative" which hasn't been clearly defined. A thumbnail image? Something with a censored section? We don't really know.

Yes I'm aware that it checks against a known database but clearly there can be collisions. So eventually it will share someone's private images.

Or rather, it will share some gray blob apparently.

Re: Hash collision in Apple NeuralHash model

#332

Post by JKCalhoun »

fsloth wrote: post #70
FabHK wrote: post #18

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6.…

"How can you use it for targeted attacks?" Just insert a known CSAM image on target's device. Done. I presume this could be used against a rival political party to ruin their reputation - insert bunch of CSAM images on their devices. "Party X is revealed as an abuse ring". This goes oh-so-very-nicely with Qanon conspiracy theories which even don't require any evidence to propagate widely. Wait for Apple to find the i…

> Just insert a known CSAM image on target's device.

Or maybe thirty. You have to surpass the threshold.

Also, if Twitter, Google, Microsoft are already deploying CSAM scanning in their services .... why are we not hearing about all the "swatting"?

Re: Hash collision in Apple NeuralHash model

#333

Post by rootusrootus »

bitneuker wrote: post #190
magpi3 wrote: post #159

Can someone ELI5? I understand that a person can now generate an image with the same hash as an illegal image (such as child porn), but I don't understand how they can get it on someone's phone and I don't understand why someone would get in trouble for an image, when finally examined, that is clearly not child pornography.

A vulnerability by itself is not that dangerous, but in combination with a sophisticated attack, or another vulnerability can be disastrous. State actors have the resources to exploit a number of unknown bugs in combination with this collision to have Apple's systems flag persons of interest. This, combined with human error during the manual review process might result in someone getting reported. Seeing as twitter (…

> State actors have the resources

You can end the conversation right there. If you are up against a state actor, you have already lost.

Re: Hash collision in Apple NeuralHash model

#334

Post by short_sells_poo »

stetrain wrote: post #258
UncleMeat wrote: post #157

Earlier quoted context omitted.

It just seems to me that there are two very different conversations happening at the same time, with people swapping back and forth between them 1. There can be false positives or other mechanisms for innocent people to get flagged. 2. It is bad to do this sort of check on the local disk. The discussion at hand started as entirely #1. But now you've swapped to #2, talking about government spying on local files. It ma…

This exactly. I am mostly convinced based on the technical details that have (slowly) come out from Apple that they have made this system sufficiently inconvenient to use as a direct surveillance system by a malicious government. Yes a government could secretly order Apple to make changes to the system, but they could also order them to just give them all of your iCloud photos and backups, or send data directly from…

Yes of course it is more private than outright scanning all photos but how does that relate to the ground state of private photos simply not being scanned at all?

And please do not bring in whataboutist arguments like "but other cloud providers are already doing it".

I'm honestly taken aback by how many people on HN are completely OK with what Apple is pulling here.

In my mind, it's irrelevant that the current system is "sufficiently inconvenient to use as a dragnet surveillance system", because it's the first step towards one that is convenient to use and if we extrapolate all the other similar efforts, we know full well what is going to happen.

Re: Hash collision in Apple NeuralHash model

#335

Post by robertoandred »

mannerheim wrote: post #184
shapefrog wrote: post #120

Earlier quoted context omitted.

I have seen it suggested that everyone should flood the system with flagged images to overwhelm it in protest to this move by apple. Sounds pretty stupid to me to fill your phone with kiddie porn in protest, but you do you internet people.

You don't need to do that, just use images that collide with the hashes.

How will you know something collides?

Re: Hash collision in Apple NeuralHash model

#336

Post by floatingatoll »

bitneuker wrote: post #190
magpi3 wrote: post #159

Can someone ELI5? I understand that a person can now generate an image with the same hash as an illegal image (such as child porn), but I don't understand how they can get it on someone's phone and I don't understand why someone would get in trouble for an image, when finally examined, that is clearly not child pornography.

A vulnerability by itself is not that dangerous, but in combination with a sophisticated attack, or another vulnerability can be disastrous. State actors have the resources to exploit a number of unknown bugs in combination with this collision to have Apple's systems flag persons of interest. This, combined with human error during the manual review process might result in someone getting reported. Seeing as twitter (…

State actors will just Gitmo you, without all this wasteful effort on hashes. This system offers no benefit sufficient to make it worth their time if they want to cull you from the population somehow.

Re: Hash collision in Apple NeuralHash model

#337

Post by atoav »

devwastaken wrote: post #306
hughrr wrote: post #7

That’s end game. Now you can use it for targeted attacks against innocent people. This needs to be shut down and disposed of immediately. There is no other outcome which is socially acceptable for Apple. I feel vindicated now. There are a lot of people saying that I’m insane as I’ve dumped the entire iOS ecosystem in the last week. But Craig was busy steamrolling out the marketing still only a couple of days back abo…

The collisions are supposedly reviewed, my concern is that the process for photodna isn't going to always be the same, and we don't actually have any knowledge as to whether their claims are true. Eventually they will phase out human intervention and replace with gameable AI. 3 letter agencies don't need to review the actual images, they can easily get federal warrants based on some numbers. Apple is content with put…

One thing about this is that we are farther and farther away from "if there is evidence for a crime a jury can understand it and rationally decide what do with it" and we are getting closer to "if this lightbulb is glowing the machine says they are guilty, so better trust us".

This kind of stuff should not be evidence, if anything it should be a indicator where to look.

Re: Hash collision in Apple NeuralHash model

#338

Post by roenxi »

tantalor wrote: post #286
ethbr0 wrote: post #248

Earlier quoted context omitted.

That's an incomplete statement. Currently, they must comply with warranty requests by scanning if they have the ability to scan . If they have no such ability (say, because they designed their phones from a privacy-first perspective), the law makes no requirement that they create such a capability. And that's what pisses people off about this.

Can a warrant compel them to develop the capability?

Have they been served a warrant?

Just because someone could force you to do something you don't want to do is not really a major concern if you choose, preemptively, to do the thing. The concerning part here is Apple signalling that they have executives that don't see phone scanning as a problem. That is a major black eye for their branding.

Re: Hash collision in Apple NeuralHash model

#339

Post by jamesdwilson »

sandworm101 wrote: post #316
tantalor wrote: post #286

Earlier quoted context omitted.

Can a warrant compel them to develop the capability?

Yes. Lavabit. Those warrants demanded that Lavabit alter its system to capture passwords and/or decrypt stored email. Lavbit instead decided to stop operating and delete everything rather than comply. Such warrants have not been fully tested in courts but they do exist.

IIRC they gave up the information AND they shut down

Re: Hash collision in Apple NeuralHash model

#340

Post by robertoandred »

HumblyTossed wrote: post #271

This can also be used to make Apple's system useless, no? If enough (millions?) of people were to, say, go to a web site and save generated gray-blobs to their phones, it would create enough false-positives to kill this system, right? Maybe game it and have everyone convert their various profile pics to these images.

And how would anyone know that those gray blobs match child porn?
Post reply on HN