Adversarial image attacks are no joke • Hacker News Board

Re: Adversarial image attacks are no joke

#171

Post by google234123 » Tue, Nov 30, 2021, 2:20 AM UTC

littlestymaar wrote: post #155

ineedasername wrote: post #149

Earlier quoted context omitted.

Hundreds you say? Think again, it's happened. (Well, 100) https://www.google.com/amp/s/www.cbc.ca/amp/1.5192656

That's a fun trivia, but that's not really relevant to the discussion here: those people took no risk for their lives when following these instructions. Making hundreds of people go in the wrong place is easy, but the difference between a bad IA and a human driver is that the majority of drivers realize when they're doing something dangerous and stop. This common sense of danger is what IAs must have before we can tr…

> A human would be like “wait, I think I saw a cyclist, slow down and figure out what this really was”.

This explains why there are only 5.25 million car accidents a year in the US.

Re: Adversarial image attacks are no joke

#172

Post by comfysocks » Tue, Nov 30, 2021, 2:22 AM UTC

mgraczyk wrote: post #169

comfysocks wrote: post #168

When we design classical control systems, the performance limitations are well understood. If we do not exceed the limits, we expect the system to be well behaved. By contrast, DNN/CNN based systems can be a bit of a black box. We can only evaluate performance empirically, not analytically. It is difficult to know where in the input space the failure modes lie. It is difficult to then build the larger system around i…

The major players are not doing it the way you describe. Tesla's driving system is not a giant model trained to imitate a human brain. There are separate perception, planning, and control algorithms.

These other algorithms would be part of the "larger system built around it" that I mention. This larger system has the task of doing what human judgement does.

Re: Adversarial image attacks are no joke

#173

Post by google234123 » Tue, Nov 30, 2021, 2:23 AM UTC

ClumsyPilot wrote: post #162

YetAnotherNick wrote: post #39

Earlier quoted context omitted.

You could spray handful of nails in the road and I think there is a big chance it would cause an accident. Or you could just dig up a hole using tools available in most homes. Agreed, it's not that easy, but not hard as well.

Yeah i think if you start digging in the middle of a busy junction, people will have some questions for you

You would think people would have questions for bike thieves using angle grinders but nope. Just throw on a high-vis yellow jacket.

Re: Adversarial image attacks are no joke

#174

Post by mgraczyk » Tue, Nov 30, 2021, 2:30 AM UTC

comfysocks wrote: post #172

mgraczyk wrote: post #169

Earlier quoted context omitted.

The major players are not doing it the way you describe. Tesla's driving system is not a giant model trained to imitate a human brain. There are separate perception, planning, and control algorithms.

These other algorithms would be part of the "larger system built around it" that I mention. This larger system has the task of doing what human judgement does.

The "larger system around it" is not trained to "do what a human brain does". That's called "behavioral cloning". The major players do not do that.

That's why the systems are more robust than you probably think to failures in perception. It's also why these systems sometimes fail in ways that humans would never fail.

Re: Adversarial image attacks are no joke

#175

Post by mgraczyk » Tue, Nov 30, 2021, 2:35 AM UTC

semi-extrinsic wrote: post #130

mabbo wrote: post #114

Earlier quoted context omitted.

> if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. This is the tricky bit. Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental st…

> So how do we decide when the automated car is 'good enough'? This is actually a really interesting point. I don't think people appreciate how far accident rates have actually dropped for modern cars without self driving. Even at million-cars-per-year sales rate you will need years of data to prove that a single self-driving software+hardware combo is better than humans with high statistical confidence. Your develop…

Fortunately waymo and others do have years of data.

Also, you can make reasonable inferences about fatal accidents using non-fatal accidents. All fatal accidents are also normal accidents. If waymo has far fewer non fatal accidents, you can reasonably infer it would have fewer fatal accidents. Otherwise you'd have to believe waymo's accidents are more likely to be fatal, but the opposite is probably true because of the locations and speeds where they drive (at least for the passenger lol)

You can also make inferences about accidents based on disengagements or undesirable events (human labeled). It's not as data limited as you might think.

Re: Adversarial image attacks are no joke

#176

Post by kingcharles » Tue, Nov 30, 2021, 2:36 AM UTC

eximius wrote: post #64

aneutron wrote: post #54

Earlier quoted context omitted.

What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the c…

> It gives the illusion of security, but they would absolutely not deter a determined threat actor. Sure. And the threat of jail/imprisonment doesn't deter determined murderer's. It doesn't mean we shouldn't put deterrents.

Absolutely we need deterrents, otherwise chaos.

What deterrents is one of the hardest problems society has ever grappled with. How do we stop antisocial behaviours? Prisons (a modern punishment) do not seem to work, for a multitude of complicated reasons. This is coming from someone who has been through the system.

Re: Adversarial image attacks are no joke

#177

Post by nemo44x » Tue, Nov 30, 2021, 2:46 AM UTC

boomboomsubban wrote: post #170

brokenmachine wrote: post #153

Earlier quoted context omitted.

>wearing it in public is endangering the public. Only because you're driving a car that was programmed by monkeys and sold by PT Barnum. If your car can't tell the difference between a street sign and a T-shirt, it's really not fully self driving, is it?

I agree that it's a ridiculous hypothetical and any company shipping something like that should also face punishment.

Only the company shipping that should face punishment. Their code instructed a machine to kill people, etc. The hypothetical t-shirt is data, not code. These are very important distinctions. Code needs to be responsible for its data.

I’d even argue that to be used on public road that any self driving code needs to be open source.

Re: Adversarial image attacks are no joke

#178

Post by astrange » Tue, Nov 30, 2021, 2:49 AM UTC

glitchc wrote: post #122

mabbo wrote: post #114

Earlier quoted context omitted.

> if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. This is the tricky bit. Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental st…

> Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental stamina' can make a big difference in safe driving. To be clear we are talking about CV which relies on passive optical sensing in the visual spectrum through cameras, not r…

> To be clear we are talking about CV which relies on passive optical sensing in the visual spectrum through cameras, not radar or lidar or IR or multi-spectral sensors.

Well, I think you mean working off RGB data? That's not necessarily the problem you have to solve even if your parts are regular cameras, as long as they're dedicated to your uses. You can modify them to see IR or polarization.

Re: Adversarial image attacks are no joke

#179

Post by comfysocks » Tue, Nov 30, 2021, 2:51 AM UTC

mgraczyk wrote: post #174

comfysocks wrote: post #172

Earlier quoted context omitted.

These other algorithms would be part of the "larger system built around it" that I mention. This larger system has the task of doing what human judgement does.

The "larger system around it" is not trained to "do what a human brain does". That's called "behavioral cloning". The major players do not do that. That's why the systems are more robust than you probably think to failures in perception. It's also why these systems sometimes fail in ways that humans would never fail.

I think you are misreading me. I am not trying to suggest they work in the same way, simply that they have the same overall task, which I consider to be a very difficult problem.

To your second point, I think we might agree that in order to be more robust to failures in perception, it would be good to understand where the failure modes live. I personally think we need a better understanding than we have today.

Re: Adversarial image attacks are no joke

#180

Post by adrianN » Tue, Nov 30, 2021, 4:31 AM UTC

aneutron wrote: post #54

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the c…

Our whole society is based on the assumption that there are very few determined threat actors. Literally nothing would function if not almost everybody would agree that it's a bad idea to try and break it. I don't think you can change that without making society an unlivable hell.