Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

101–110 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#101

Post by colpabar »

slg wrote: post #75
foerbert wrote: post #69

Earlier quoted context omitted.

The difference is the ease with which they can demur. Before, it would be a whole heck of a lot of new, additional work. They also have the problem of actually introducing it without being noticed, or having to come up with some cover for the new behavior. Now? Well now it's real simple. It will even conveniently not expose the actual images it's checking for. Apple now has significantly less ability to rationally re…

As far as I'm aware, this system is not new. It is only moving from the cloud to the local device. If the cloud was already compromised, which it seems like it would be in your logic since all the same reasoning applies, I don't understand the complaints about it moving locally. In my mind there are two possible ways to view this. We could trust Apple last month and we can trust them today. We couldn't trust Apple la…

>It is only moving from the cloud to the local device.

But isn't that exactly why this is such a big deal? It sets a precedent that it's ok that devices are scanning your local device for digital contraband. Sure, right now it's only for photos that are going to be uploaded to iCloud anyway. But how long before it scans everything, and there's no way to opt out?

I don't see this as so much a question of apple's trustworthiness, I see it as a major milestone in the losing battle for digital privacy. I don't think it will be long before these systems go from "protecting children from abuse" total government surveillance, and it's particularly egregious that it's being done by apple, given their previous "commitment to privacy".

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#102

Post by kook_throwaway »

slg wrote: post #75
foerbert wrote: post #69

Earlier quoted context omitted.

The difference is the ease with which they can demur. Before, it would be a whole heck of a lot of new, additional work. They also have the problem of actually introducing it without being noticed, or having to come up with some cover for the new behavior. Now? Well now it's real simple. It will even conveniently not expose the actual images it's checking for. Apple now has significantly less ability to rationally re…

As far as I'm aware, this system is not new. It is only moving from the cloud to the local device. If the cloud was already compromised, which it seems like it would be in your logic since all the same reasoning applies, I don't understand the complaints about it moving locally. In my mind there are two possible ways to view this. We could trust Apple last month and we can trust them today. We couldn't trust Apple la…

>It is only moving from the cloud to the local device.

That's the point. Yesterday someone posted a fantastic TSA metaphor where they are doing the same scans and patdowns but with agents permanently stationed in the privacy of your home where they pinkie promise it will only be before a trip to the airport and only checking the bags you will be flying with.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#103

Post by breuleux »

riffraff wrote: post #86
breuleux wrote: post #67

Earlier quoted context omitted.

Presumably Apple would be afraid that, say, the EU becomes suspicious, issues a court order to obtain the hashes, notices they cannot audit the CCP hashes, pointedly asks "what is this", becomes absolutely livid that their citizens are spied on by a country that is not them, fines Apple out the wazoo, then extradites whoever is responsible and puts them in prison. I mean, China's not the only player in this. Putting…

Seems pretty trivial to have different servers sides per country, and put there a different db. EU, China, Iran, US: everyone gets to spy on their own children and forbid whatever they want.

The db is encrypted and uploaded to user devices. If each country gets a different db, the payload will be different in each country, which does not make sense if it's all supposed to be CSAM. So Apple would likely just say "these were mandated by the US government for US citizens," punting the ball in their court, unless they are forbidden to say so, in which case they'll say nothing, but we all know what it means. That's when you know you should change phones and stop using all cloud services, because obviously all cloud services scan for the same thing.

On the flip side, though, at least Apple will have given us a canary. And that's why I don't think Apple will be asked to add these hashes: if the governments don't want their citizens to know what's being scanned server side, pushing the equivalent data to clients would tip their hand. They might just write Apple off as a loss and rely on Google, Facebook, etc.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#104

Post by yeldarb »

nanis wrote: post #84

It sounds like there is code that enables anyone to compute the perceptual hash for an image. Is this code published somewhere?

The instructions are here: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX

This was discussed on HN yesterday here (and a few other front-page stories): https://news.ycombinator.com/item?id=28218391

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#105

Post by nybble41 »

rootusrootus wrote: post #78
Dah00n wrote: post #53

Earlier quoted context omitted.

If they pass CSAM verfied by hash on to human verification inside Apple they break the law. Not even the FBI are allowed to do that. Only NMCEC is an allowed recipient by US federal law.

Seems to be a misunderstanding between what the law appears to say and what the actual practice is. Law enforcement's interest is not served by trying to prosecute moderators or companies acting in good faith because they have CSAM in their possession.

There is a difference between moderators manually identifying illegal content in a stream of mostly-legal material and a process where content which has already been matched against the database and classified as almost-certainly-illegal is subjected to further internal review.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#106

Post by WesolyKubeczek »

endisneigh wrote: post #8

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’…

You know, it's rather not okay to treat every smartphone user out there as a potential criminal just because they happen to have photos on their devices. At least in an alleged democracy where there's this presumption of innocence thing.

Even Pegasus wasn't that much rotten, it at least wasn't indiscriminately installed onto everyone's phone.

But what can I say, there wasn't much uproar about piracy tax on blank CD-R(W) media back in the day, so why not have that now. And eventually we go peak USSR where everyone and their dog is suspect and whoever is arrested is the enemy of the people. Yay, it's somehow reassuring to know I won't live long enough to see it.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#107

Post by Youden »

endisneigh wrote: post #8

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’…

> It doesn’t matter if there are collisions if the two images don’t actually look the same.

Is that really true? My understanding is that the manual reviewers at Apple only see some kind of low-resolution proxy, not the full-resolution image. I'd also be shocked if the human reviewers were shown the original, actually CP image, to compare to.

Given that, it's not necessary to produce an actual visual match, it's just necessary to produce an image that when scaled-down looks like CSAM (e.g. take an actual photo of a kid at the beach and photoshop in some skin-coloured swimwear with creases in the right places).

> Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America?

The attack I'd worry about here is similar to swatting. Someone who doesn't like me sends a bunch of images like the ones I described above (not just one), they end up synced to iCloud (because Apple wants _everything_ synced to iCloud) and Apple reports me to the authorities, who end up knocking at my door and arresting me.

Even though I'm innocent, I'll probably have most of my computers confiscated for a while and spend a few days locked up.

> PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of.

PhotoDNA's algorithms and hashes aren't public, so it's not clear how an attacker would exploit PhotoDNA in the way that people are afraid will be done for Apple.

PhotoDNA also isn't, as far as I know, part of a product that aims to create unprotected backups of the phones of nearly a billion users. Apple really wants you to upload your whole phone to iCloud. The only comparable alternative is Google's Android backup but Google does the right thing and end-to-end encrypts that.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#108

Post by smithza »

yeldarb wrote: post #98
smithza wrote: post #91

This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. If the author was comparing 2 trillion pictures of people, or children specifically, I think this false-posi…

The sample set is ImageNet, which is a well-known dataset in Computer Vision and is available for download here: https://www.kaggle.com/c/imagenet-object-localization-challe... I'd love to see this work extended; if you find additional collisions in the wild please submit a PR to the repo (please do not submit artificially generated adversarial images): https://github.com/roboflow-ai/neuralhash-collisions For what it…

    For what it's worth, Apple claimed to find a _lower_ incidence of false-positives when it used pornographic images in its test[1] (which makes sense; images containing humans is probably more aligned with what the model was trained on than nematodes)
This is an important note. Is it the case that this algorithm is trained for humans or not? the 1/trillion false-positive rate might imply it is trained with a broader set.

Thank you for those helpful tidbits.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#109

Post by [deleted] »

orhmeh09 wrote: post #54
stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Could you provide specific evidence that China has and would do this? I’ve a hard time recalling any specific cases. Maybe nation-states do this kind of thing, but I’m only aware of the countless times the United States has done this. What’s the recent history?

[deleted]

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#110

Post by yongjik »

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

These scenarios sound rather like "the wrong side of airlock" stories[1]. Why would China go through an elaborate scheme with fake child-porn hashes, when it can already arrest these people on made-up charges, and simply tell Apple to provide the private key for their phones, so that they can read and insert whatever real/fake evidences they want?

[1] I'm stealing the expression from this excellent article: https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...

Post reply on HN