Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

441–450 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#441

Post by syshum »

incrudible wrote: post #395
csmpltn wrote: post #242

Earlier quoted context omitted.

> "The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it." This can be abused to spam Apple's manual review process, grinding it down to a halt. You've cost Apple time and money by making them review each such fake report.

> You've cost Apple time and money by making them review each such fake report. Ok, but… how do I profit? If I wanted to waste Apple employee time, I could surely find a way to do it, but why would I? The functioning of society relies on the fact that people generally have better things to do than waste each others time.

>>Ok, but… how do I profit?

To some people the ethical problems with this system are worth the effort to make their implementation impossible.

if you are asking for monetary profit, that does not have to be a goal for someone to want to engage in disruption of this service.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#442

Post by rustymonday »

cat199 wrote: post #7
alfalfasprout wrote: post #2

There's been a lot of focus on the likelihood of collisions and whether someone could upload eg; an image with a matching hash to your device to "set you up", etc. But what's still extremely concerning is that there is still no guarantee that the hash list used can't be coopted for another purpose (eg; politically insensitive content).

this also punts the debate to the checking process instead of the fact that there even is a process to start with..

I'm so glad you said this.

As someone not in the tech field, it is incredibly concerning that half the people here on Hacker News, people who help build this kind of technology, do not seem to be concerned with what Apple is doing.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#443

Post by wbl »

Notanothertoo wrote: post #325
stefan_ wrote: post #282

Earlier quoted context omitted.

You are one underpaid random guy in India looking at CSAM all day clicking the wrong button away from a raid of your home and the end of your life as you know it.

This, these charges are damning once they are made. Plus the countless legal dollars you are going to have to front and hours spent proving innocence and that's assuming the justice system actually works.. Try explain this to your employer while you start missing deadlines due to court dates.. The police also could easily leverage this to warrant hop. As they have been found doing in the past. I think the bike rider…

A false report is not going to lead to charges. In order to bring a case they need evidence of possession.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#444

Post by sennight »

skygazer wrote: post #420
giantrobot wrote: post #254

Earlier quoted context omitted.

It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a legitimate CSAM image and modify it such that when scaled for use in the perceptual algorithm(s) it changes to be some politically sensitive image. Upon review it'll look like CSAM so off it goes to reporting agencies. Because the perceptual hash algorithms are…

This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it. Granted, Apple makes no mention of any safeg…

The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review.

You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database.

Note its NeuralHash.

Find a non-CP image that would, after being scaled down or otherwise sanitized, fool an unaccountable and likely disinterested Apple employee into muttering "close enough" while selecting whichever option box it is that causes life ruination.

Feed that imagine into an adversarial network until it spits out the desired NeuralHash.

Distribute that image to everyone who has ever disagreed with you on the internet, prayed to the wrong god, competed with you in business, voted the wrong way, etc.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#445

Post by nr2x »

endisneigh wrote: post #8

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’…

"If some corrupt government wants to get you they don’t need this"

-> if some corrupt government wants to *FIND YOU SHARING IDEAS THEY DON'T LIKE* they can *DEFINITELY* use this

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#446

Post by syshum »

cheald wrote: post #384
mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

Why would anyone bother calling the cops and telling them that someone they don't like is an imminent threat? The end result is that some officer just has to stop by and see that they aren't actually building bombs. You've cost someone a bit of time, but that's it.

Reality shows that not to be the case. How many "SWATTING" incidents now has resulted in much more than "some officer stopping by"....

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#447

Post by partdavid »

madeofpalk wrote: post #426
rovr138 wrote: post #339

Earlier quoted context omitted.

On top of that, what happens if a court/government orders them to give them all the current data about people with matches, regardless of the 30 matches. They can't say if it is or not a match so they have to go after the individuals. Is that enough evidence for a warrant? Someone in the court thinks it's true and can't prosecute?, oh, it got leaked . -- Not every country has the same protections about innocent until…

My understanding is that these "safety vouchers" are uploaded regardless of a match. Only when there is about 30 matches are those safety vouches able to be decrypted to determine there there was a match. So Apple claims your threat model is not technically possible. Besides, Govt. can just order Apple to hand over the photos themselves from iCloud Photos because those are not end-to-end encrypted.

Which is an individual and legal process, i.e., it requires a search warrant. There are certainly problems with this process, but at a minimum, an account needs to be already and individually identified through some process (suspicion) and a legal process vetted by a judge happens (probable cause) to allow the access.

That's not at all the same as proactively casting a net and starting an investigation based on the results.

Apple has designed the system so that 30 matches are required; they could include more key material in each safety voucher to reduce the number required, or make it only require one match by providing the whole key in each voucher, or forego the system entirely in favor of one without such restrictions (which they can do, given some time, with an iOS update). It isn't "not technically possible" it's just "how they designed it", which is what the poster is saying Congress would ask about.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#448

Post by heavyset_go »

theluketaylor wrote: post #229
zug_zug wrote: post #184

Earlier quoted context omitted.

And what if the FBI demands a list of all Apple users who have matched even 1 CSAM photo for their own private watchlist?

There are a lot of really valid criticisms of Apple plan here, but Apple has gone out of their way to prevent that exact case. Apple is using secret splitting to make sure they cannot decode the CSAM ticket until the threshold is reached. Devices also produce some synthetic matches to prevent themselves Apple (or anyone else) inferring a pre-threshold count based on the number of vouchers. https://www.apple.com/child…

> There are a lot of really valid criticisms of Apple plan here, but Apple has gone out of their way to prevent that exact case.

Not they haven't. The files are still uploaded to Apple's servers where Apple holds their encryption keys. Apple or the FBI could scan the photos for CSAM whenever they want to.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#449

Post by jachee »

sennight wrote: post #444
skygazer wrote: post #420

Earlier quoted context omitted.

This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it. Granted, Apple makes no mention of any safeg…

The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after…

… and repeat “on the order of 30+“ times to trigger the reporting threshold, and convince them to save it into their synced-to-iCloud Photos library.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#450

Post by cinquemb »

sennight wrote: post #444
skygazer wrote: post #420

Earlier quoted context omitted.

This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it. Granted, Apple makes no mention of any safeg…

The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after…

> The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review.

Yeah, mentioned something like that here [0]:

>> And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this.

[0] https://news.ycombinator.com/item?id=28177170

Post reply on HN