Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

191–200 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#191

Post by btilly »

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do?

I think that the real risk is, "put these hashes in your database or we will stop all iPhone related manufacturing in China".

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#192

Post by mrtranscendence »

zbobet2012 wrote: post #16
endisneigh wrote: post #8

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’…

That's not the route in which this will be exploited, and used at scale. A corrupt government has to know they want you "to just get you". Instead they will embed a collision in anti-government meme. That collision will flag you, and now they know you harbor doubts and will come get you. This is why it's a privacy concern. It's no the tech (like you said photo dna's been about forver), it's the scanning of the phone.

> That collision will flag you, and now they know you harbor doubts and will come get you.

Only if that government has worked out some deal with Apple whereby such an anti-government meme would end in the government being notified accordingly. Don't forget that you need a sufficiently high number of collisions, for one, and that those collisions are audited by Apple before being sent to law enforcement.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#193

Post by colpabar »

slg wrote: post #113
colpabar wrote: post #101

Earlier quoted context omitted.

> It is only moving from the cloud to the local device. But isn't that exactly why this is such a big deal? It sets a precedent that it's ok that devices are scanning your local device for digital contraband. Sure, right now it's only for photos that are going to be uploaded to iCloud anyway. But how long before it scans everything, and there's no way to opt out? I don't see this as so much a question of apple's trus…

>But how long before it scans everything, and there's no way to opt out? Do we think this is detectable? If yes, then why worry about it if we will know when this switch is made? If not, why did we trust Apple that this wasn't happening already? That is the primary thing I don't understand, this fear rests on an assumption that Apple is a combination of both honest and corrupted. If they are honest, we have no reason…

It feels like you're viewing this as a purely hypothetical question and ignoring reality. No company is 100% good or bad, and it doesn't make any sense to force all possible interpretations into good/bad.

>If not, why did we trust Apple that this wasn't happening already?

I do not trust Apple. I don't really trust any major tech company, because they put profit first, and everything else comes second. I believe that a company as large as apple is already colluding with government(s) to surveil people, because if a money-making organization is offered a government contract that involves handing over already collected data, and it was kept secret by everyone involved, why would they refuse? I know that's very cynical, but I can't see it any other way.

But that's beside the point, which is that what apple is doing is paving the way for normalization of mass government surveillance on devices that we're supposed to own.

>If they were corrupted, why tell us about this at all?

So that we can all get used to it, and not make a big fuss when google announces android will do the same thing. It's much easier to do things without needing to keep them a secret. This is in no way only about apple, they're just breaking the ice so to speak.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#194

Post by e40 »

mrtranscendence wrote: post #179
saithound wrote: post #163

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino…

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

> The end result is that some peon at Apple has to look at the images and mark them as not CSAM

Btw, this reminded me of a podcast about FB's group to do just this. Because it negatively impacted the mental health of those FB employees, they farmed it out to the contractors in other countries. There were interviews with women in the Philippines, and it was having the same impact there.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#195

Post by nybble41 »

shadowfiend wrote: post #119
nybble41 wrote: post #96

Earlier quoted context omitted.

That's too little, too late. By the time any human reviewer can evaluate whether the images should have been in the database the encryption on the backups has already been circumvented and other parties have already been given access to your private files.

That is not how this works. Please read up on the functioning of the system before chiming in with such certainty on its behavior. The only thing they will have gained access to are the “derivatives” (presumably lower res versions) of the matched photos, which if this is done to frame you is strictly the fake CSAM.

I'm aware of how the system works. Those are still your private files which were supposed to be encrypted and which were revealed (either partially or fully, makes no difference) to another party. That fact that this review process is even possible means that the encryption on the backups can't be trusted.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#196

Post by ctur »

The more interesting point about hash collisions is probably less about accidental clashes and more about intentional clashes. If the hashes in the CSAM database were known publicly, and people began generating intentional clashes with innocuous images, and those images were on many phones, it could basically DOS whatever manual process Apple creates.

Basically it could become an arms race where, say, free speech advocates convince people to just keep and share some images and overwhelm the process. Then Apple adapts to new hashes, blocklists known false positives, and the cycle repeats.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#197

Post by mrtranscendence »

zug_zug wrote: post #184
endisneigh wrote: post #8

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’…

And what if the FBI demands a list of all Apple users who have matched even 1 CSAM photo for their own private watchlist?

And what if the FBI demands that Apple leadership genuflect toward an oil painting of J Edgar Hoover? The FBI can demand all sorts of things. In this case, Apple isn't tracking collisions as small as 1 photo.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#198

Post by vineyardmike »

btilly wrote: post #191
stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do? I think that the real risk is, "put these hashes in your database or we will stop all iPhone related manufacturing in China".

Both would be bad for apple. Chinese sales are already significant enough to their bottom line to comply.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#199

Post by [deleted] »

captn3m0 wrote: post #115
stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Related, the Indian Government (Telecom Department) bullied Apple into building an iOS feature for reporting phone calls and SMS by threatening to stop iPhone sales in India. Apple complied. https://indianexpress.com/article/technology/mobile-tabs/app...

[deleted]

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#200

Post by sharken »

sigmar wrote: post #143
slg wrote: post #113

Earlier quoted context omitted.

>But how long before it scans everything, and there's no way to opt out? Do we think this is detectable? If yes, then why worry about it if we will know when this switch is made? If not, why did we trust Apple that this wasn't happening already? That is the primary thing I don't understand, this fear rests on an assumption that Apple is a combination of both honest and corrupted. If they are honest, we have no reason…

Are we going to need to reverse engineer every single Apple update to make sure the feature hasn't creeped into non-iCloud uses? Is the inevitable Samsung version of this system going to be as privacy-preserving? How are we sure the hash list isn't tainted? All of these problems are solved by one principle: Don't put the backdoor code in the OS to begin with.

At this point it will be the OS without on-device scanning that wins for me.

It's amazing that Apple is still thinking about moving this idea forward, but they must think they are untouchable and can do what they want.

And it adds insult to injury that two very different images can generate the same NeuralHash value.

Post reply on HN