Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

111–120 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#111

Post by sillysaurusx »

breuleux wrote: post #103
riffraff wrote: post #86

Earlier quoted context omitted.

Seems pretty trivial to have different servers sides per country, and put there a different db. EU, China, Iran, US: everyone gets to spy on their own children and forbid whatever they want.

The db is encrypted and uploaded to user devices. If each country gets a different db, the payload will be different in each country, which does not make sense if it's all supposed to be CSAM. So Apple would likely just say "these were mandated by the US government for US citizens," punting the ball in their court, unless they are forbidden to say so, in which case they'll say nothing, but we all know what it means.…

This isn’t true. The db is blinded. We have no way of knowing what’s in it.

It would be trivial to have the same payload on each device, and extract different answers using the matching server side db which varies by country.

Perhaps not trivial, but just short.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#112

Post by headShrinker »

rootusrootus wrote: post #32
minitoar wrote: post #11

Earlier quoted context omitted.

The damage is already done by the time it gets to the point of devices being confiscated.

By the time the FBI comes knocking for your devices, they have a lot of evidence, not a list of hash collisions.

> By the time the FBI comes knocking for your devices, they have a lot of evidence, not a list of hash collisions.

Not necessarily. The FBI knocks on your door when they have convinced a warrant-signing judge, that’s there is probable-cause, or that additional information can be collected to build a case in which the defendant will ‘plead’ or make a deal for sentence reduction. 90% of defendants make a deal and never go to trial because by that time the stakes are so high a guilty verdict includes the harshest possible sentence.

FBI only needs to convince you they can win a case or the judge that there might be fruit behind a locked door, they don’t need direct evidence. It’s really up to the judge.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#113

Post by slg »

colpabar wrote: post #101
slg wrote: post #75

Earlier quoted context omitted.

As far as I'm aware, this system is not new. It is only moving from the cloud to the local device. If the cloud was already compromised, which it seems like it would be in your logic since all the same reasoning applies, I don't understand the complaints about it moving locally. In my mind there are two possible ways to view this. We could trust Apple last month and we can trust them today. We couldn't trust Apple la…

> It is only moving from the cloud to the local device. But isn't that exactly why this is such a big deal? It sets a precedent that it's ok that devices are scanning your local device for digital contraband. Sure, right now it's only for photos that are going to be uploaded to iCloud anyway. But how long before it scans everything, and there's no way to opt out? I don't see this as so much a question of apple's trus…

>But how long before it scans everything, and there's no way to opt out?

Do we think this is detectable? If yes, then why worry about it if we will know when this switch is made? If not, why did we trust Apple that this wasn't happening already?

That is the primary thing I don't understand, this fear rests on an assumption that Apple is a combination of both honest and corrupted. If they are honest, we have no reason to distrust what they are saying about how this system functions or will function in the future. If they were corrupted, why tell us about this at all?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#114

Post by yusefnapora »

endisneigh wrote: post #15
bawolff wrote: post #13

Earlier quoted context omitted.

If the two images looked the same, then the expected behaviour is a collision, so if collisions matter at all, it would only be for pictures that look different.

They don’t matter because if two images don’t look the same, but collide - then human processes will absolve you. This isn’t some AI that sends you straight to prison lol

Imagine this scenario.

- You receive some naughty (legal!) images of a naked young adult while flirting online and save them to your camera roll.

- These images have been made to collide [1] with "well known" CSAM images obtained from the dark underbelly of the internet, on the assumption that their hashes will be contained in the encrypted database.

- Apple's manual review kicks in because you have enough such images to trigger the threshold.

- The human reviewer sees a bunch of thumbnails of naked people whose age is indeterminate but looks to be on the young side.

- Your case is forwarded to the FBI, who now have cause to turn your life upside down.

This scenario seems entirely plausible to me, given the published information about the system and the ability to generate collisions that look like an arbitrary input image, which is clearly possible as demonstrated in the linked thread. The fact that most of us are unlikely to be targets of this kind of attack is little comfort to those that may be.

[1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#115

Post by captn3m0 »

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Related, the Indian Government (Telecom Department) bullied Apple into building an iOS feature for reporting phone calls and SMS by threatening to stop iPhone sales in India.

Apple complied.

https://indianexpress.com/article/technology/mobile-tabs/app...

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#116

Post by axelf4 »

yongjik wrote: post #110
stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

These scenarios sound rather like "the wrong side of airlock" stories[1]. Why would China go through an elaborate scheme with fake child-porn hashes, when it can already arrest these people on made-up charges, and simply tell Apple to provide the private key for their phones, so that they can read and insert whatever real/fake evidences they want? [1] I'm stealing the expression from this excellent article: https://d…

That is not a good comparison. The extra hashes would help China find out about more borderline citizens than it otherwise would have.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#117

Post by burkaman »

yongjik wrote: post #110
stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

These scenarios sound rather like "the wrong side of airlock" stories[1]. Why would China go through an elaborate scheme with fake child-porn hashes, when it can already arrest these people on made-up charges, and simply tell Apple to provide the private key for their phones, so that they can read and insert whatever real/fake evidences they want? [1] I'm stealing the expression from this excellent article: https://d…

Because they don't know who to arrest yet. The idea isn't to fabricate a charge, it's to locate people sharing politically sensitive images that the government hasn't already identified.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#118

Post by nonbirithm »

WesolyKubeczek wrote: post #106
endisneigh wrote: post #8

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’…

You know, it's rather not okay to treat every smartphone user out there as a potential criminal just because they happen to have photos on their devices. At least in an alleged democracy where there's this presumption of innocence thing. Even Pegasus wasn't that much rotten, it at least wasn't indiscriminately installed onto everyone's phone. But what can I say, there wasn't much uproar about piracy tax on blank CD-R…

And it's fine to treat everyone as a potential criminal when we entrust our data on the same company's servers? No matter if on-device scanning makes surveillance easier than ever, the surveillance itself was still a significant possibility up to this point with server-side scanning. Imagine how many petabytes of user data already exist in the cloud.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#119

Post by shadowfiend »

nybble41 wrote: post #96
tehnub wrote: post #51

Earlier quoted context omitted.

Apple did mention in their security thread model document [0] this: Apple will also refuse all requests to instruct human reviewers to file reports for anything other than CSAM materials for accounts that exceed the match threshold. [0]: https://www.apple.com/child-safety/pdf/Security_Threat_Model...

That's too little, too late. By the time any human reviewer can evaluate whether the images should have been in the database the encryption on the backups has already been circumvented and other parties have already been given access to your private files.

That is not how this works. Please read up on the functioning of the system before chiming in with such certainty on its behavior.

The only thing they will have gained access to are the “derivatives” (presumably lower res versions) of the matched photos, which if this is done to frame you is strictly the fake CSAM.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#120

Post by stickfigure »

endisneigh wrote: post #87
stickfigure wrote: post #42

Earlier quoted context omitted.

Your browser cache is not being synced to iCloud; Apple doesn't see it. So no, it is not currently possible.

I'm not talking about Apple, I'm saying in general technology has already been deployed to make what you're describing possible. So where's the evidence of abuse?

No vendors are snooping your phone/computer browser cache, so this attack vector does not yet exist. Apple is building it.
Post reply on HN