Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

411–420 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#411

Post by danudey »

hnechochamber wrote: post #392
JunkDNA wrote: post #383

Earlier quoted context omitted.

Bingo. I wish I could upvote this comment more. All the geeks get distracted by words like “cloud” or “virtual” and forget that all this stuff we depend on has a physical presence at some point in the real world. That physical presence necessitates humans interacting with other humans. Humans interacting with humans falls squarely in the “things governments poke their noses into” bucket. It’s like the early days of N…

Dont worry, ill fix all this by creating a unique javascript framework that will change the world.

Maybe we could make this framework future-proof by using blockchains? Somehow? Maybe it can use blockchains, or it can be stored on a blockchain, or maybe both at the same time. Surely that will help society in some nonspecific, ambiguous manner.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#412

Post by mdoms »

mrtranscendence wrote: post #179
saithound wrote: post #163

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino…

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

What if I can generate an attack that will mark your own picture of your own toddler nude in a bathtub as CSAM? Do you still feel confident in "some peon at Apple" to mark it as not CSAM?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#413

Post by dang »

solidgol wrote: post #408

I think the headline is misleading. The point of the article is that the algorithm that Apple is using works as well as Apple says it does, and the headline implies that it doesn't.

What would be a better (i.e. more accurate and neutral) title?

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#414

Post by zymhan »

hnechochamber wrote: post #392
JunkDNA wrote: post #383

Earlier quoted context omitted.

Bingo. I wish I could upvote this comment more. All the geeks get distracted by words like “cloud” or “virtual” and forget that all this stuff we depend on has a physical presence at some point in the real world. That physical presence necessitates humans interacting with other humans. Humans interacting with humans falls squarely in the “things governments poke their noses into” bucket. It’s like the early days of N…

Dont worry, ill fix all this by creating a unique javascript framework that will change the world.

No, we just need another anonymous distributed networking/storage/socialmedia/coffeemaker protocol to save us

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#415

Post by orange_puff »

What we know from this well written and helpful article; The false positive rate Apple told us their algorithm had seems to be accurate. If a machine learning model is extracted from the OS it exists on, it will be much easier to generate adversarial attacks.

For example, a neural net's cost function is just a multivariate function with weights as its input. To figure out how to move those weights (positively or negatively), the gradient of the function is calculated and the weights are nudged in the opposite direction (gradient is the direction of the largest growth of a function, we are trying to minimize the cost). Now, assume we are given a cost function and the weights are constant, now, the input can be the image. So, we take the gradient of the cost function with respect to the image pixels and can now see how we should nudge those to maximize the cost. Apple will absolutely need to protect against adversarial attacks for this to be viable. I'm hopeful.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#416

Post by danudey »

t0mas88 wrote: post #332
c7DJTLrn wrote: post #322

Earlier quoted context omitted.

I'm so done. I'm sorry to dump a pointless rant like this on HN but... what the hell is going on these days? Nobody seriously seems to care about legitimate privacy concerns anymore. If I were in a position of power, like being CEO, CTO, or even just an engineer on the team at Apple that implemented this, I'd do EVERYTHING to make sure that my power is in check and that I'm not pushing a fundamentally harmful technol…

I'm as surprised as you are that a giant like Apple doesn't just tell them "go ahead, ban iPhones, see how popular they'll become" to someone as powerless as the government of India. It would be a huge free publicity campaign for them in the rest of the world while the public in India would either put pressure on their government or buy iPhones via import websites. For additional fun, strike a deal with the #2 non go…

I wonder how Apple's shareholders would react if the company threw away a market that was worth $1.8bn in revenue in 2020.

Then there's China; 17% of Apple's global revenue, $43.7bn. I don't think shareholders would much appreciate that.

> the public in India would either put pressure on their government or buy iPhones via import websites

The iPhone had 2.97% market share in India in April 2021, down from a high of 3.54% in June 2020. I don't think the people who wanted to buy iPhones but couldn't would be able toput any significant amount of political pressure on politicians.

Rich people would just import them from somewhere else like they always have before, and everyone else would switch to some available Android phone that had the modifications that the government wanted.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#417

Post by AnthonyMouse »

danudey wrote: post #409
AnthonyMouse wrote: post #402

Earlier quoted context omitted.

> All along these people thought algorithms and protocols (e.g. bitcoin and TCP/IP) would somehow be a powerful force that would cause governments to fall on their knees and let people evade government control. After all, it's distributed! You can't stop it! But that's the underlying problem here. Apple isn't a standardized protocol or a distributed system. It's a monolithic chokepoint. You can't do this with a PC. D…

Dell ships laptops with tons of Dell software, as well as tons of third-party software. Do you really think that, if they wanted to, they couldn't just update one of those pieces of software to enable remote installs? Hell, Dell has shipped more than one bug that allowed attackers administrator-level access or worse, I wouldn't put it past them to come up with some kind of asinine feature that not only lets them push…

> Do you really think that, if they wanted to, they couldn't just update one of those pieces of software to enable remote installs?

If they did that, people would wipe their device and not reinstall the Dell software. Many people do this already as soon as they buy the device.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#418

Post by Spooky23 »

saithound wrote: post #299
ChrisKnott wrote: post #286

Earlier quoted context omitted.

You're adding quite a lot of technobabble gloss to an "attack vector" that boils down to "people can send you images that are visually indistinguishable from known CSAM". Guess what, they can already do this but worse by just sending you actual illegal images of 17.9 year olds. While it would be bad to be subjected to such an attack, and there is a small chance it would lead to some kind of interaction with law enfor…

I suggest you reread the comment, because "people can send you images that are visually indistinguishable from known CSAM" is not what is being said at all. Where did you even get that from? The point is precisely that people can become victims of various new attacks, without ever touching photos that are actual "known CSAM". For Christ's sake, half the comments here are about how adversaries can create and spread po…

> No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people.

An attacker can embed a matching image inside of a PowerPoint zip file, and email it to any corporate employee using O365.

Or, an angry parent can call the police and let them know that a 16 year old possesses nose pictures of their 15 year old girlfriend.

The over top response to this controversy is really disappointing.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#419

Post by k2enemy »

vineyardmike wrote: post #264
mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

> The end result is that some peon at Apple has to look at the images and mark them as not CSAM. As others said, if the non-csam looks sexual at all, they'll probably get flagged for post-apple review. Beyond that, it doesn't seem to be in apple's interest to be conservative in flagging. An employee reviewer's best interest is to minimize false negatives not false positives. As many mentioned, even an investigation c…

> Beyond that, it doesn't seem to be in apple's interest to be conservative in flagging. An employee reviewer's best interest is to minimize false negatives not false positives.

I would have thought the opposite. If there is a false positive that leads to an arrest and ruins someone's life, but the public sees that it is a false positive, then Apple will take an enormous hit in the marketplace. Nobody will want to take on the demonstrated real risk being falsely accused of possessing CSAM.

If they have a false negative, it is unclear to me what negative effects Apple would suffer. As far as I know, nobody would know about it outside of Apple.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#420

Post by skygazer »

giantrobot wrote: post #254
mrtranscendence wrote: post #179

Earlier quoted context omitted.

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a legitimate CSAM image and modify it such that when scaled for use in the perceptual algorithm(s) it changes to be some politically sensitive image. Upon review it'll look like CSAM so off it goes to reporting agencies. Because the perceptual hash algorithms are…

This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it.

Granted, Apple makes no mention of any safeguard, but it would be trivial in principle to protect against, and is not an unavoidable failing.

Post reply on HN