Adversarial image attacks are no joke • Hacker News Board

Re: Adversarial image attacks are no joke

#161

Post by d110af5ccf » Tue, Nov 30, 2021, 1:44 AM UTC

mgraczyk wrote: post #59

ReleaseCandidat wrote: post #49

Earlier quoted context omitted.

> We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. Actually no. We know that only some psychopaths would do that and so the risk is minimal. AI is currently simply not 'good enough' to be used in critical environments. The problem is that _any_ sticker or even dirt or snow or ... on any road sign can lead to misinterpretation, you can…

Sometime iceberg lettuce kills people (salmonella). We have safety regulations and inspections to mitigate that, but you can never prove that iceberg lettuce is safe.

This is specious. I can take actions to mitigate food poisoning such as rinsing my vegetables prior to consumption. If a company claims that their leafy greens are already rinsed and safe to consume without further prep work when in fact they are not then presumably they can be held liable.

The risks surrounding vegetable packaging and distribution are well understood, readily quantifiable, and possible to mitigate.

Computer vision algorithms on the other hand are poorly understood black boxes with seemingly arbitrary failure modes. We do not (yet) appear to understand how to quantify or mitigate the associated risks. The consequences of failure are quite severe in comparison to food poisoning. Only a fool would trust their life to them.

Re: Adversarial image attacks are no joke

#162

Post by ClumsyPilot » Tue, Nov 30, 2021, 1:46 AM UTC

YetAnotherNick wrote: post #39

marcosdumay wrote: post #31

Earlier quoted context omitted.

I'm pretty sure this would fail to kill people on almost every place you could try it. And if it works somewhere, it's because there are other problems with the road that should be fixed. Human driving is full of redundancies, and there is a clear hierarchy of information. People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say. If your automated driving system doesn…

You could spray handful of nails in the road and I think there is a big chance it would cause an accident. Or you could just dig up a hole using tools available in most homes. Agreed, it's not that easy, but not hard as well.

Yeah i think if you start digging in the middle of a busy junction, people will have some questions for you

Re: Adversarial image attacks are no joke

#163

Post by d110af5ccf » Tue, Nov 30, 2021, 1:49 AM UTC

JoshuaDavid wrote: post #75

goatlover wrote: post #44

Earlier quoted context omitted.

> Why not just actually swap the road sign? Because you have to physically do it, as opposed to hacking from anywhere else on the planet. > not on a system that has a more general sense of collision-avoidance and situational awareness (like humans do). Are vision systems to that point yet when it comes to driving vehicles? > Because the whole point of using an automated vision system is usually that you want to avoid…

> Because you have to physically do it, as opposed to hacking from anywhere else on the planet. My impression is that the adversarial image attacks in question involve physically placing a sticker on something which will be in the view of self-driving cars -- it's not a remote exploit.

Which crime is easier to commit - physically swapping a street sign or placing a sticker on an existing one? How long does each act take? What equipment do you have to carry on you for each task?

In a given span of time, how many street signs can a single person swap out versus how many stickers can they apply?

When sourcing the materials for an attack, how expensive are stickers relative to physical signs?

Re: Adversarial image attacks are no joke

#164

Post by d110af5ccf » Tue, Nov 30, 2021, 1:54 AM UTC

NavinF wrote: post #86

pixelgeek wrote: post #16

Earlier quoted context omitted.

I don't think that the word 'train' should be used for these systems. We feed then reams of data and effectively cull the ones that don't work but the critical problem is that we judge the effectiveness of an ML system and we actually do know what the ML systems is supposed to be looking for. We feed a system a series of images of bikes and then select the ones that can pick out a bike but we don't know how the bike…

Thank you for an accurate ELI5 description of the human visual system. Dunno what this “ML” is, I assume it’s some part of the brain? It’s too bad you can’t analyze brains like you can with neural networks. It’s trivial to visualize filters and feature maps or to create heatmaps showing which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?).

The issue is that a human driver is much more than just a visual cortex.

> which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?)

Sure, you can watch the Rube Goldberg machine work. It doesn't mean you understand why it works on a conceptual level or have any hope of rigorously quantifying when and how it could fail.

Re: Adversarial image attacks are no joke

#165

Post by [deleted] » Tue, Nov 30, 2021, 1:54 AM UTC

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

[deleted]

Re: Adversarial image attacks are no joke

#166

Post by mlac » Tue, Nov 30, 2021, 1:57 AM UTC

d110af5ccf wrote: post #163

JoshuaDavid wrote: post #75

Earlier quoted context omitted.

> Because you have to physically do it, as opposed to hacking from anywhere else on the planet. My impression is that the adversarial image attacks in question involve physically placing a sticker on something which will be in the view of self-driving cars -- it's not a remote exploit.

Which crime is easier to commit - physically swapping a street sign or placing a sticker on an existing one? How long does each act take? What equipment do you have to carry on you for each task? In a given span of time, how many street signs can a single person swap out versus how many stickers can they apply? When sourcing the materials for an attack, how expensive are stickers relative to physical signs?

What’s the point of the attack? A well-placed sticker for the first car that comes along with a high value target? Just kill that person an easier way, that is more certain.

After one accident, the sticker will be removed.

Industrial sabotage to take out one car’s camera system? Ok - but which car company will do that? It’s mutually assured destruction if the other actors retaliate, and serious legal fees if caught.

High school pranks? Sure. But again, they will be identified, finger printed, the printed item will be analyzed and reviewed for which printer printed it, and the person will be ID’d.

Re: Adversarial image attacks are no joke

#167

Post by mannerheim » Tue, Nov 30, 2021, 2:04 AM UTC

mgraczyk wrote: post #160

d110af5ccf wrote: post #157

Earlier quoted context omitted.

To be fair, obscenity laws don't seem (to me) to be in keeping with the rest of the laws surrounding freedom of speech in the US. They are shockingly vague and have been used to prosecute people for artwork (!!!) in the past. Court verdicts for such laws seem to be mixed - sometimes they go along with them, sometimes they side with freedom of expression. > offers to engage in illegal transactions Just to clarify, tha…

I'm not misunderstanding, although I agree that all this is largely beside the point. It's not obviously true that the illegality of possession automatically implies the illegality of the offer to sell. Consider for example a case where the seller doesn't actually possess the item, but merely offers to sell it. In this case, the supreme court upheld a law making such offers illegal. Without the ruling, you might argu…

Is there a law against teaching someone how to make nuclear weapons? There are certainly export controls, and almost anyone with the knowledge is subject to NDAs with the government, but are there any actual laws criminalising telling people how to make nuclear weapons?

Re: Adversarial image attacks are no joke

#168

Post by comfysocks » Tue, Nov 30, 2021, 2:14 AM UTC

When we design classical control systems, the performance limitations are well understood. If we do not exceed the limits, we expect the system to be well behaved. By contrast, DNN/CNN based systems can be a bit of a black box. We can only evaluate performance empirically, not analytically. It is difficult to know where in the input space the failure modes lie. It is difficult to then build the larger system around it, because you do not know were the keep-out zones are.

I think a study of the failure modes of CNNs shouldn't be interpreted as an all-or-nothing evaluation of the technology as a whole, but rather a step towards gaining some confidence regarding its reliability. A lot more work needs to be done before I will trust it to drive my car.

Regarding the use of CNN's for autonomous driving, I think it is insane that people are trying to do this by trying to solve a VERY hard problem, i.e. making a machine that can do what the human brain does. Your neural net does not have enough labels to account for all possible scenarios. Instead, it would make more sense to redesign the infrastructure in a way that bounds the problem space. The current system is designed for human drivers. We should make a system that is easy to interpret for both human and machine drivers. Of course this infrastructure would benefit all car makers, not just the first mover.

Re: Adversarial image attacks are no joke

#169

Post by mgraczyk » Tue, Nov 30, 2021, 2:17 AM UTC

comfysocks wrote: post #168

When we design classical control systems, the performance limitations are well understood. If we do not exceed the limits, we expect the system to be well behaved. By contrast, DNN/CNN based systems can be a bit of a black box. We can only evaluate performance empirically, not analytically. It is difficult to know where in the input space the failure modes lie. It is difficult to then build the larger system around i…

The major players are not doing it the way you describe. Tesla's driving system is not a giant model trained to imitate a human brain. There are separate perception, planning, and control algorithms.

Re: Adversarial image attacks are no joke

#170

Post by boomboomsubban » Tue, Nov 30, 2021, 2:19 AM UTC

brokenmachine wrote: post #153

boomboomsubban wrote: post #116

Earlier quoted context omitted.

> The physical act of owning an explosive can be made illegal and is. Then let me change my example to show legal items being used with the intent to cause harm is still illegal. I'm free to put razors into candy, but if I hand it out on Halloween it'd be illegal. >However, it would be an extreme stretch to apply or expand these to apply to simply wearing a t-shirt. There is already plenty of case law that says weari…

>wearing it in public is endangering the public. Only because you're driving a car that was programmed by monkeys and sold by PT Barnum. If your car can't tell the difference between a street sign and a T-shirt, it's really not fully self driving, is it?

I agree that it's a ridiculous hypothetical and any company shipping something like that should also face punishment.