Adversarial image attacks are no joke • Hacker News Board

Re: Adversarial image attacks are no joke

#181

Post by boomboomsubban » Tue, Nov 30, 2021, 5:01 AM UTC

nemo44x wrote: post #177

boomboomsubban wrote: post #170

Earlier quoted context omitted.

I agree that it's a ridiculous hypothetical and any company shipping something like that should also face punishment.

Only the company shipping that should face punishment. Their code instructed a machine to kill people, etc. The hypothetical t-shirt is data, not code. These are very important distinctions. Code needs to be responsible for its data. I’d even argue that to be used on public road that any self driving code needs to be open source.

>Their code instructed a machine to kill people, etc. The hypothetical t-shirt is data, not code.

It is data you are intentionally using to hurt someone. If a person legally gained access to a water treatment plant and sent inputs to poison the water, you wouldn't say that person only sent data and the code was at fault. The shirt is inputting data into the car the same way.

Tools should be made with safety in mind, but when people use them to cause harm the person is at fault too.

Re: Adversarial image attacks are no joke

#182

Post by brokenmachine » Tue, Nov 30, 2021, 5:16 AM UTC

boomboomsubban wrote: post #181

nemo44x wrote: post #177

Earlier quoted context omitted.

Only the company shipping that should face punishment. Their code instructed a machine to kill people, etc. The hypothetical t-shirt is data, not code. These are very important distinctions. Code needs to be responsible for its data. I’d even argue that to be used on public road that any self driving code needs to be open source.

>Their code instructed a machine to kill people, etc. The hypothetical t-shirt is data, not code. It is data you are intentionally using to hurt someone. If a person legally gained access to a water treatment plant and sent inputs to poison the water, you wouldn't say that person only sent data and the code was at fault. The shirt is inputting data into the car the same way. Tools should be made with safety in mind,…

If I made a computer that explodes and catches fire when you type certain words into it, is it then the typists fault if that word gets typed?

Or have I been negligent by creating a device that is unable to process certain words without exploding? ie it can't do it's job properly and safely.

This hypothetical car is too badly designed to ignore things that aren't road signs.

A 5 year old can recognize if something is a road sign or not, and we don't let them drive.

That's how pathetic this car is.

Of course it shouldn't be allowed on a public road, and it shouldn't be allowed to be sold because it is unsafe.

Re: Adversarial image attacks are no joke

#183

Post by boomboomsubban » Tue, Nov 30, 2021, 6:09 AM UTC

brokenmachine wrote: post #182

boomboomsubban wrote: post #181

Earlier quoted context omitted.

>Their code instructed a machine to kill people, etc. The hypothetical t-shirt is data, not code. It is data you are intentionally using to hurt someone. If a person legally gained access to a water treatment plant and sent inputs to poison the water, you wouldn't say that person only sent data and the code was at fault. The shirt is inputting data into the car the same way. Tools should be made with safety in mind,…

If I made a computer that explodes and catches fire when you type certain words into it, is it then the typists fault if that word gets typed? Or have I been negligent by creating a device that is unable to process certain words without exploding? ie it can't do it's job properly and safely. This hypothetical car is too badly designed to ignore things that aren't road signs. A 5 year old can recognize if something is…

>Of course it shouldn't be allowed on a public road, and it shouldn't be allowed to be sold because it is unsafe

I agree with this, that's separate from what I'm discussing.

If you made a computer that exploded when someone typed a word into it, and then I knew about this and told someone else to type that word in, we would both be culpable.

Re: Adversarial image attacks are no joke

#184

Post by josefx » Tue, Nov 30, 2021, 6:42 AM UTC

xyzzyz wrote: post #78

aneutron wrote: post #54

Earlier quoted context omitted.

What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the c…

Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the compound in it. Gaining access is rather easy. You can easily fly drones over most of reservoirs and dump whatever you want into them. Making strong poisons is also relatively easy, eg. dimethylmercury can be easily synthesized by any chemistry graduate.

This results in a dead chemistry graduate.

Re: Adversarial image attacks are no joke

#185

Post by littlestymaar » Tue, Nov 30, 2021, 9:06 AM UTC

google234123 wrote: post #171

littlestymaar wrote: post #155

Earlier quoted context omitted.

That's a fun trivia, but that's not really relevant to the discussion here: those people took no risk for their lives when following these instructions. Making hundreds of people go in the wrong place is easy, but the difference between a bad IA and a human driver is that the majority of drivers realize when they're doing something dangerous and stop. This common sense of danger is what IAs must have before we can tr…

> A human would be like “wait, I think I saw a cyclist, slow down and figure out what this really was”. This explains why there are only 5.25 million car accidents a year in the US.

Most of them being benign collisions though. “Only” 34k of them are fatal. For like 270M vehicle in circulation.

I don't know how many automated vehicle Uber has, but with one killed already the ratio is a few orders of magnitude worse for their automated vehicle than for the average driver in the US. (probably even way worse than the average drunk driver actually)

Re: Adversarial image attacks are no joke

#186

Post by Isinlor » Tue, Nov 30, 2021, 1:01 PM UTC

genewitch wrote: post #70

Isinlor wrote: post #30

Earlier quoted context omitted.

There is plenty of natural "adversarial examples" to worry about. Like billboard with stop sign on it. https://youtu.be/-OdOmU58zOw?t=149

I'll be more inclined to start believing that self driving / autonomous vehicles are actually "coming soon" when the federal government decrees it is illegal to wear clothing with certain markings/colors. No red octogons, no reflective red and white parts, no yellow vertical stripes, etc. I don't think that "cause an air to fail to stop" is the correct threat to address, I think "making AI stop and therefore cause tr…

My prediction is that it will happen with 50% confidence before/after 2029. Or 50% confidence that it will be between 2026 and 2031.

Basically they need to improve their driving software some 10 000x times. From driving 100km before safety critical disengagement to 1 million kilometers. 1 - 2 million milles is benchmark presented by CJ Moore, Tesla’s director of autopilot software to California Department of Motor Vehicles.

> “Tesla is at Level 2 currently. The ratio of driver interaction would need to be in the magnitude of 1 or 2 million miles per driver interaction to move into higher levels of automation. Tesla indicated that Elon is extrapolating on the rates of improvement when speaking about L5 capabilities. Tesla couldn’t say if the rate of improvement would make it to L5 by end of calendar year.”

If they manage to keep on doubling distance driven every 6 months then we should be there in:

log2(10000) * 6 months = 8 years

You can make your own predictions here: https://www.metaculus.com/questions/5304/widely-available-te...

Re: Adversarial image attacks are no joke

#187

Post by pixelgeek » Tue, Nov 30, 2021, 3:12 PM UTC

d110af5ccf wrote: post #164

NavinF wrote: post #86

Earlier quoted context omitted.

Thank you for an accurate ELI5 description of the human visual system. Dunno what this “ML” is, I assume it’s some part of the brain? It’s too bad you can’t analyze brains like you can with neural networks. It’s trivial to visualize filters and feature maps or to create heatmaps showing which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?).

The issue is that a human driver is much more than just a visual cortex. > which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?) Sure, you can watch the Rube Goldberg machine work. It doesn't mean you understand why it works on a conceptual level or have any hope of rigorously quantifying when and how it could fail.

I really like the Goldberg machine analogy. Consider it stolen

Re: Adversarial image attacks are no joke

#188

Post by ravi-delia » Tue, Nov 30, 2021, 3:56 PM UTC

d110af5ccf wrote: post #164

NavinF wrote: post #86

Earlier quoted context omitted.

Thank you for an accurate ELI5 description of the human visual system. Dunno what this “ML” is, I assume it’s some part of the brain? It’s too bad you can’t analyze brains like you can with neural networks. It’s trivial to visualize filters and feature maps or to create heatmaps showing which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?).

The issue is that a human driver is much more than just a visual cortex. > which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?) Sure, you can watch the Rube Goldberg machine work. It doesn't mean you understand why it works on a conceptual level or have any hope of rigorously quantifying when and how it could fail.

I don't think that's entirely fair. I'd bet that with a reasonably sized team, good introspective tools, and several years, you could reverse engineer what each part of a network does. Of course that's only a gut feeling, and there'd be no way of proving it was accurate.

Re: Adversarial image attacks are no joke

#189

Post by rstuart4133 » Tue, Nov 30, 2021, 10:03 PM UTC

eximius wrote: post #64

aneutron wrote: post #54

Earlier quoted context omitted.

What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the c…

> It gives the illusion of security, but they would absolutely not deter a determined threat actor. Sure. And the threat of jail/imprisonment doesn't deter determined murderer's. It doesn't mean we shouldn't put deterrents.

For what it's worth, the threat of jail / imprisonment has no effect on determined murders whatsoever. In fact it has no effect violent crimes in general. From https://www.vera.org/downloads/publications/for-the-record-p... :

> The weak association between higher incarceration rates and lower crime rates applies almost entirely to property crime.16 Research consistently shows that higher incarceration rates are not associated with lower violent crime rates.

It does make sense. If a person is committing a crime in the hopes of material gain, reducing that material gain by imposing a negative gain if they get caught should deter them.

It doesn't seem like a person committing a crime of passion would be using that sort calculus. And it turns out in this case intuition is right: the figures say they don't. Ergo the threat of jail has no effect on the number of murders committed.

Re: Adversarial image attacks are no joke

#190

Post by brokenmachine » Tue, Nov 30, 2021, 10:10 PM UTC

boomboomsubban wrote: post #183

brokenmachine wrote: post #182

Earlier quoted context omitted.

If I made a computer that explodes and catches fire when you type certain words into it, is it then the typists fault if that word gets typed? Or have I been negligent by creating a device that is unable to process certain words without exploding? ie it can't do it's job properly and safely. This hypothetical car is too badly designed to ignore things that aren't road signs. A 5 year old can recognize if something is…

>Of course it shouldn't be allowed on a public road, and it shouldn't be allowed to be sold because it is unsafe I agree with this, that's separate from what I'm discussing. If you made a computer that exploded when someone typed a word into it, and then I knew about this and told someone else to type that word in, we would both be culpable.

>and told someone else to type that word in, we would both be culpable.

Perhaps as some technicality but in reality who is (much) more culpable? Who would think a computer would be made so poorly that it fails dangerously if the wrong word is typed in?

One could say they didn't believe it would be true because it sounds too implausible.

Same with a car that drives off the road if it sees the wrong thing on a t-shirt.

I don't think it's fine for a company to just make products that are super dangerous if some slight edge case is met. In my opinion they would be liable.

Otherwise terrorism can be legal if you slap an "AI" label on it.