Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ImageNet contains naturally occurring Apple NeuralHash collisions

blog.roboflow.com

Post reply on HN
Print view View original

401–410 of 530 posts

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#401

Post by mLuby »

runawaybottle wrote: post #364
captn3m0 wrote: post #115

Earlier quoted context omitted.

Related, the Indian Government (Telecom Department) bullied Apple into building an iOS feature for reporting phone calls and SMS by threatening to stop iPhone sales in India. Apple complied. https://indianexpress.com/article/technology/mobile-tabs/app...

I think few people are making the appropriate parallel. What we’re looking at is not necessarily government overreach, but fascism. When the hell did it become Apple’s job to do this? Apple is not a branch of law enforcement. The government needs warrants for stuff like this. We are merging corporate and government interests here. Repeat after me, Apple is not supposed to be a branch of law enforcement. It also says…

> When the hell did it become Apple’s job to do this?

Apple provided a pathway, however unintentionally, to greater power. And those in power used their existing authority to gather even more for themselves, as they always do.

Like drops flow into streams into rivers into oceans, power aggregates at the top until regime change spills it back to the ground.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#402

Post by AnthonyMouse »

rsj_hn wrote: post #357
c7DJTLrn wrote: post #322

Earlier quoted context omitted.

I'm so done. I'm sorry to dump a pointless rant like this on HN but... what the hell is going on these days? Nobody seriously seems to care about legitimate privacy concerns anymore. If I were in a position of power, like being CEO, CTO, or even just an engineer on the team at Apple that implemented this, I'd do EVERYTHING to make sure that my power is in check and that I'm not pushing a fundamentally harmful technol…

What is going on is that reality is slapping some techno-utopians in the face and they are shocked, shocked, that governments are more powerful that businesses. That's not at all what the lefty geeks learned by reading Chomsky or what the righty geeks learned by reading Heinlein. All along these people thought algorithms and protocols (e.g. bitcoin and TCP/IP) would somehow be a powerful force that would cause govern…

> All along these people thought algorithms and protocols (e.g. bitcoin and TCP/IP) would somehow be a powerful force that would cause governments to fall on their knees and let people evade government control. After all, it's distributed! You can't stop it!

But that's the underlying problem here. Apple isn't a standardized protocol or a distributed system. It's a monolithic chokepoint.

You can't do this with a PC. Dell and HP don't retain the ability to push software to hardware they don't own after they've already sold it and against the will of the person who does own it.

People pointed out that this would happen. Now it's happening. Qué sorpresa.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#403

Post by avereveard »

tablespoon wrote: post #214
stjohnswarts wrote: post #122

Earlier quoted context omitted.

Have we established that a US NGO is accepting "CSAM" hashes from China or that they are cooperating with them at all? That seems unlikely and Apple hasn't yet announced plans with how they're going to scan phones in China, I mean wouldn't China just demand outright to have full scanning capabilities of anything on the phone since you don't have any protection at all from that in China?

> Have we established that a US NGO is accepting "CSAM" hashes from China or that they are cooperating with them at all? I believe Apple's intention is to accept hashes from all governments, not just one US organization. One of their ineffectual concessions to the criticism was to require two governments provide the same hash before they'd start using it.

China can definitely find a state government requiring some cash injection to help push the hash of a certain uninteresting square where nothing happened into the db

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#404

Post by mLuby »

incrudible wrote: post #400
saithound wrote: post #343

Earlier quoted context omitted.

Yes, I can. This is just one possible strategy: there are many others, where different things are done, and where things are done in a different order. You use the collider [1] and one of the many scaling attacks ([2] [3] [4], just the ones linked in this thread) to create an image that matches the hash of a reasonably fresh CSAM image currently circulating on the Internet, and resizes to some legal sexual or violent…

This leaves open the question of how the image gets on the device of the victim. You would have to craft a very specific image that the victim is likely to save, and the existence of such a specially crafted file would completely exonerate them.

I'd think the attack would be done in reverse:

1. Get a photo that the target already has.

2. Generate an objectionable image with the same hash as the target's photo. (This is obviously illegal.)

3. Submit the objectionable image to the government database.

Now the target's photo will be flagged until manually reviewed.

This doesn't sound impossible as a targeted attack, and if done on a handful of images that millions of people might have saved (popular memes?) it might even grind the manual reviews to a halt. But maybe I'm not understanding something in this (very bad idea) system.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#405

Post by jrockway »

stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Why does the CCP even need to talk to Apple? They have a database of CSAM, and can modify an image in this set to collide with a special image they're looking for on people's phones. They then share their new modified cache of CSAM with other countries ("hey, China is helping us! that's great!") and it gets added to Apple's database for the next iOS release, because it looks to humans like CSAM. Only the CCP knows that it collides with something special they're looking for.

Now that we know that collisions are not just easy -- but happen by themselves with no human input (as evidenced by the Imagenet collisions), we know this system can't work. Apple has two remaining safeguards: a second set of hashes (they say), and human reviewers.

The human reviewers are likely trained to prefer false-positives when unsure, and so while a thorough human review would clearly indicate "not CSAM" for the images the malicious collisions match, it doesn't feel like much of a safeguard to me. (Remember the leaked memo from the US's organization -- they called our objections "screeching voices". I'm sure the actual reviewers think similarly.)

I assume the people reviewing CSAM for the CCP will be in China, so they can be in on the whole scheme. (In the US, we have slightly better checks and balances. Eventually the image will be in front of a court and a jury of your peers, and it's not illegal to have photos that embarrass the CCP, so you'll personally be fine modulo the stress of a criminal investigation. But that dissident in China, probably not going to get a fair trial -- despite the image not being CSAM, it's still illegal. And Apple handed it directly to the government for you.)

I don't know, I just find this thing exceedingly worrying. When faced with a government-level adversary, this doesn't sound like a very good system. I think if we're okay with this, we might as well mandate CCTV in everyone's home, so we can catch real child abusers in the act.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#406

Post by danudey »

captn3m0 wrote: post #115
stickfigure wrote: post #19

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in…

Related, the Indian Government (Telecom Department) bullied Apple into building an iOS feature for reporting phone calls and SMS by threatening to stop iPhone sales in India. Apple complied. https://indianexpress.com/article/technology/mobile-tabs/app...

They added a feature which is off by default and allows a user to select a supported installed app to use as a spam reporting app.

IMHO this is great, I wish more countries would enable this feature. Something like 95% of my phone calls are spam, to the point where I just don't answer the phone anymore unless they're in my contacts list. Users being able to actually report them as spam might actually result in this BS finally stopping.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#407

Post by vmladenov »

runawaybottle wrote: post #373
vmladenov wrote: post #323

Earlier quoted context omitted.

> So Apple scans Chinese (or whoever) citizens libraries, finds "CSAM" and reports them to ICMEC which then reports them to the government in question. If Apple finds that a particular hash is notorious for false positives, they can reject it / ask for a better one. And they’re not scanning your library; it’s a filter on upload to iCloud. The FUD surrounding this is getting ridiculous.

Look, I said it in another post, it is not Apple’s job to act as an arm of law enforcement. The same way it is not either of our jobs to be vigilante sheriffs and police the streets. We’re talking about a company that makes phones and computers, and sells music and tv shows via the internet. Does that matter at all? How about this. All car manufacturers must now wirelessly transmit when the driver of the car is speed…

It becomes their job when you ask them to host your images on their property. If you don’t, then there’s nothing happening.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#409

Post by danudey »

AnthonyMouse wrote: post #402
rsj_hn wrote: post #357

Earlier quoted context omitted.

What is going on is that reality is slapping some techno-utopians in the face and they are shocked, shocked, that governments are more powerful that businesses. That's not at all what the lefty geeks learned by reading Chomsky or what the righty geeks learned by reading Heinlein. All along these people thought algorithms and protocols (e.g. bitcoin and TCP/IP) would somehow be a powerful force that would cause govern…

> All along these people thought algorithms and protocols (e.g. bitcoin and TCP/IP) would somehow be a powerful force that would cause governments to fall on their knees and let people evade government control. After all, it's distributed! You can't stop it! But that's the underlying problem here. Apple isn't a standardized protocol or a distributed system. It's a monolithic chokepoint. You can't do this with a PC. D…

Dell ships laptops with tons of Dell software, as well as tons of third-party software. Do you really think that, if they wanted to, they couldn't just update one of those pieces of software to enable remote installs?

Hell, Dell has shipped more than one bug that allowed attackers administrator-level access or worse, I wouldn't put it past them to come up with some kind of asinine feature that not only lets them push new software/drivers/whatever to the machine, but lets attackers do so as well.

Re: ImageNet contains naturally occurring Apple NeuralHash collisions

#410

Post by danudey »

rsj_hn wrote: post #357
c7DJTLrn wrote: post #322

Earlier quoted context omitted.

I'm so done. I'm sorry to dump a pointless rant like this on HN but... what the hell is going on these days? Nobody seriously seems to care about legitimate privacy concerns anymore. If I were in a position of power, like being CEO, CTO, or even just an engineer on the team at Apple that implemented this, I'd do EVERYTHING to make sure that my power is in check and that I'm not pushing a fundamentally harmful technol…

What is going on is that reality is slapping some techno-utopians in the face and they are shocked, shocked, that governments are more powerful that businesses. That's not at all what the lefty geeks learned by reading Chomsky or what the righty geeks learned by reading Heinlein. All along these people thought algorithms and protocols (e.g. bitcoin and TCP/IP) would somehow be a powerful force that would cause govern…

> All along these people thought algorithms and protocols (e.g. bitcoin and TCP/IP) would somehow be a powerful force that would cause governments to fall on their knees and let people evade government control. After all, it's distributed! You can't stop it!

The internet and its design and associated protocols were designed to work around external forces - a nuclear attack or natural disaster. It was never designed to be government-proof. People who thought that would be the case were being idealistic and naive.

If you want real change in the world, as you said, you have to affect the political world, which is an option available to any citizen or corporation who can spend millions on lobbyists.

Post reply on HN