Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adversarial image attacks are no joke

unite.ai

Post reply on HN
Print view View original

111–120 of 196 posts

Re: Adversarial image attacks are no joke

#111

Post by ghaff »

aneutron wrote: post #103
eximius wrote: post #100

Earlier quoted context omitted.

Generally calling something security theatre has an implication that it shouldnt be done because of its inefficacy and the availability of robust alternatives (e.g., port knocking is theatre when we can have robust security on known ports with minimal configuration and cryptography).

While I do agree that security theater does have a connotation for things that have no reason to be done, I only meant that it's not enough. It's theater in the sense that it would only provide a sense of safety, not solve the actual underlying issue or vulnerability class.

In general, very little is ever enough to completely prevent some sort of determined targeted attack, especially if the attacker doesn't care whether they're caught or not.

Re: Adversarial image attacks are no joke

#112

Post by KingMachiavelli »

boomboomsubban wrote: post #104
nemo44x wrote: post #98

Earlier quoted context omitted.

Explosives, poisons, drugs aren’t speech. Printing the chemistry for then is protected speech. If I wanted to print an image and put it on a t-shirt that would trick a computer driven car into doing something if its cameras saw my shirt, that’s not my problem. The barrier to entry is much lower too so I think it’s up to the engineers to solve it instead of trying to dump the hard problems on society.

This is like saying "if I set up a movement based explosive in a public place, and you just happened to walk by it, that's not my problem." Yes it is, you took actions that you knew could severely harm people.

No those are completely different. The physical act of owning an explosive can be made illegal and is. In the US the act of owning and expressing an element of speech is protected under US law.

You are getting close to something with you second statement. There are laws that criminalize actions like yelling 'Fire' inside a movie theater or provoking a fight (fighting words). Essentially these laws isolate the protected 'speech' from a non-speech and therefore non-protected 'action'.

However, it would be an extreme stretch to apply or expand these to apply to simply wearing a t-shirt. There is already plenty of case law that says wearing/displaying symbols or profanity is not enough to be considered fighting words/act. Heck, in most cases just using a racial epithet is not enough to be considered fighting words and/or hate speech. [1]

At most you will ever be able to convict is if someone is installing these adversarial images on public property (e.g street signs). In that case you might be able to use the harmful nature/intent of the images to elevate what would otherwise be a vandalism charge to assault. Essentially there needs to be a distinct and meaningful 'action' beyond just wearing/expressing speech.

[1] https://www.msn.com/en-us/news/us/federal-court-saying-the-n...

Re: Adversarial image attacks are no joke

#113

Post by KingMachiavelli »

OldHand2018 wrote: post #40

I see a completely different attack vector here. Lawyers. If you are selling a product or service that has been trained on a dataset that contains copyrighted photos you don't have permission to use and I can "prove it" enough to get you into court and into the discovery phase, you are screwed. I'll get an injunction that shuts you down while we talk about how much money you have to pay me. And lol, if any of those p…

IIRC simply training on copyrighted material is completely fine or at least you can claim fair use. As long as the market of the copyrighted material is not 'AI data training set' then it should be OK. Essentially scraping images from the internet is OK but using a pirated copyrighted commercial AI data training set is not. (Fair use doesn't necessarily exclude use for a commercial/sold product.)

But if the AI model just spits out copyrighted material verbatim then that is still owned by the actual copyright holder.

Re: Adversarial image attacks are no joke

#114

Post by mabbo »

glitchc wrote: post #102
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

Adversarial examples don't confuse people, only algorithms. Perhaps you need to face the fact that if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. I don't trust my life to your "It kinda looks like a road, oh wait it's a pylon, I've been tricked, BAM!" dumpster fire of an algorithm. We used to have to craft robustness i…

> if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world.

This is the tricky bit.

Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental stamina' can make a big difference in safe driving.

But then there will be the occasional situation in which the CV screws up, and there's an accident. Some of those are ones where many/most humans could have handled the situation better and avoided the accident.

So how do we decide when the automated car is 'good enough'? Do we have to reach a point where in no situation could any human have done better? Must it be absolutely better than all humans, all the time? Because we may never reach that point.

And all the while, we could be avoiding a lot more accidents (and deaths) from situations the AI could have handled.

Re: Adversarial image attacks are no joke

#115

Post by igorkraw »

I work in this field, I have a project specifically on adversarial examples and I have a strong opinion on this. I personally think worrying about adversarial examples in real life production systems is like worrying about getting the vanilla linux kernel to perform RT critical tasks. It is fundamentally not a burden you should put on that one component alone and is a problem you can only solve with a system approach. And if you do that, it is for all practical purposes already solved: apply multiple, random perturbations to the input, project your perturbed version onto a known,safe image space, and establish consensus. [1] is a work from my university which I like to point towards. Yes this lower accuracy, yes you won't be able to do critical things with this anymore but that's the price you pay for safety. Not getting hyped about CNNs and adopting a fail-safe approach that is only augmented with NNs is (in my humble opinion) why Waymo has 30k miles between disengagements [2] now while Tesla is either going to make me eat this post (not impossible given Andrej Karpathy is much smarter than me) OR are trying to hide the fact that they will never have anything resembling FSD by avoiding to report numbers.

[3] is another paper I recommend for anyone wanting to USE CNNs for applications and wants to calmly assess the risk associated with adversarial examples

Now, from a research perspective they are fascinating, they highlight weaknesses in our ability to train models,are a valuable tool to train robust CV models in the low data regime and have paved the way towards understanding the types of features learned in CNNs (our neighbours just released this [4] which in my eyes debunked a previously held assumptions that CNNs have a bias towards high frequency features, which is a fascinating result).

But for anyone wanting to use the models, you shouldn't worry about them because you shouldn't be using the models for anything critical in a place where an attack can happen anyway. The same way that "what is the best way to encrypt our users passwords so they cannot be stolen" is the wrong way to approach passwords "how can we make the deep neural network in the application critical path robust against targeted attack" is (for now) the wrong way to approach CV.

[1] https://arxiv.org/abs/1802.06806

[2] https://www.forbes.com/sites/bradtempleton/2021/02/09/califo...

[3]https://arxiv.org/abs/1807.06732

[4] https://proceedings.neurips.cc/paper/2020/hash/1ea97de85eb63...

Re: Adversarial image attacks are no joke

#116

Post by boomboomsubban »

KingMachiavelli wrote: post #112
boomboomsubban wrote: post #104

Earlier quoted context omitted.

This is like saying "if I set up a movement based explosive in a public place, and you just happened to walk by it, that's not my problem." Yes it is, you took actions that you knew could severely harm people.

No those are completely different. The physical act of owning an explosive can be made illegal and is . In the US the act of owning and expressing an element of speech is protected under US law. You are getting close to something with you second statement. There are laws that criminalize actions like yelling 'Fire' inside a movie theater or provoking a fight (fighting words). Essentially these laws isolate the protec…

> The physical act of owning an explosive can be made illegal and is.

Then let me change my example to show legal items being used with the intent to cause harm is still illegal. I'm free to put razors into candy, but if I hand it out on Halloween it'd be illegal.

>However, it would be an extreme stretch to apply or expand these to apply to simply wearing a t-shirt. There is already plenty of case law that says wearing/displaying symbols or profanity is not enough to be considered fighting words/act.

This hypothetical T-shirt isn't comparable to fighting words, wearing it would unquestionably cause harm to the relevant ones who encounter it. Owning or creating it might not be a crime, but wearing it in public is endangering the public.

Re: Adversarial image attacks are no joke

#117

Post by tshaddox »

aneutron wrote: post #54
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the c…

It’s pretty easy for a single person to modify or remove an important street sign. Some kids stole traffic signs and were convicted of manslaughter when someone ran a (missing) stop sign and killed another driver. https://www.washingtonpost.com/archive/politics/1997/06/21/3...

Re: Adversarial image attacks are no joke

#118

Post by gh0std3v »

aneutron wrote: post #54
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the c…

> What you are proposing are what I think would be called a security theater.

I don't think putting people to prison for, say, flipping a Tesla by screwing with its computer vision algorithm is security theatre. Rather, it's accountability. I'm pretty sure most people are aware that you cannot stop a determined attacker from breaking a system (which is exactly why Spectre mitigations were implemented as soon as the vulnerability was discovered: it's hard to exploit, but still possible).

Defining a legal code for exploiting computer systems through their hardware or their software is not security theatre, it's to ensure that we have a system to punish crime.

Re: Adversarial image attacks are no joke

#119

Post by tshaddox »

p_j_w wrote: post #89
eximius wrote: post #64

Earlier quoted context omitted.

> It gives the illusion of security, but they would absolutely not deter a determined threat actor. Sure. And the threat of jail/imprisonment doesn't deter determined murderer's. It doesn't mean we shouldn't put deterrents.

>It doesn't mean we shouldn't put deterrents. GP doesn't say we shouldn't, but rather that it's not good enough.

Depends what you mean by “not good enough.” It’s obviously not perfect, like all our laws and systems for preventing crimes.

Re: Adversarial image attacks are no joke

#120

Post by dogleash »

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

There would be support to outlaw adversarial attacks towards self-driving cars. As other posters have suggested it probably already is illegal, or is a narrow expansion of scope for existing laws.

>We can try to make them hard to make, hard to posses anonymously, and hard to deliver.

To stretch your own analogy, I have a wide selection of poisons at home. Except we call them cleaning products, insecticide and automobile fluids.

You can get public support against adversarial attacks on self-driving. Except the main use case for computer vision is passive surveillance. Good luck on that front.

Oh, and just for funzies, I'll point out the irony that some of the people building CV surveillance systems would post on HN that regardless of regulation it'll exist no matter what the government wants. The argument was that it'd be so hard for the government to control CV surveillance, that law wouldn't prevent business from creating and using it anyway. When it comes to adversarial attacks, it seems more likely to involve actions of private individuals rather than businesses, and businesses minimize legal risk in a way individual citizens don't.

Post reply on HN