Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adversarial image attacks are no joke

unite.ai

Post reply on HN
Print view View original

51–60 of 196 posts

Re: Adversarial image attacks are no joke

#51

Post by xboxnolifes »

jimbob45 wrote: post #33
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

>It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. I used to think this until someone walked me through the logistics of both and made me realize that you would need an agency-alerting level of poison for the water supply and some way to avoid people just shutting off the A/C and sticking their heads out of windows (also a huge amount of g…

> agency-alerting level of poison

Doesn't this fall under: "We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison"?

And the rest of it being you should use something odorless / tasteless.

Re: Adversarial image attacks are no joke

#52

Post by m3kw9 »

I can propose a non-trivial solution to these problems, that is to have a data cleaner average and ignore certain data like how humans does it. Humans would ignore everything else but the face and maybe the body, and also we don’t examine someone’s follicles either, we basically average.

Re: Adversarial image attacks are no joke

#53

Post by Vetch »

Ozzie_osman wrote: post #38
marcosdumay wrote: post #31

Earlier quoted context omitted.

I'm pretty sure this would fail to kill people on almost every place you could try it. And if it works somewhere, it's because there are other problems with the road that should be fixed. Human driving is full of redundancies, and there is a clear hierarchy of information. People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say. If your automated driving system doesn…

You put too much faith in humans. Things like stop sign removal have caused deaths in the past. https://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-...

Humans are not so bad as drivers. Your example is an event from over 2 decades ago and was deemed newsworthy. Humans drive in all kinds of conditions but death rate is about 1 per 100 million miles driven. A search reveals crashes to be on the order of hundreds of collisions per 100 million miles driven. Age, country, intoxication level, road design and laws, road and environmental conditions also play a major role such that accident rates for someone aged 30+ in a Northern European country are going to be a lot less than teenagers in a country where road laws are merely friendly suggestions (and considering the chaos of driving in those countries, the rates are actually surprisingly low).

Re: Adversarial image attacks are no joke

#54

Post by aneutron »

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

What you are proposing are what I think would be called a security theater.

It gives the illusion of security, but they would absolutely not deter a determined threat actor.

The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the compound in it. It's unpractical even for "underground" types. Especially the quantities required.

Mathematics and computer science is a different story in my opinion. You cannot restrict science or thought. You can try, but good luck. The most you can do is delay it. If there is an attack that enables someone to flip a Tesla on the road (as suggested below), the security theater will hide the attack from common folk, but determined actors will reach it eventually, and at that point, they can deploy it as they wish. And in contrast to the water plant, the logistical endeavor to exploit it is absolutely easy in comparison: slap a sticker on your vehicle.

Security by obscurity or by theater is rarely a good strategy in my opinion. We should absolutely be transparent about these kind of things, and allow researchers full access to develop attacks against these systems, and effectively communicate when they are found.

Re: Adversarial image attacks are no joke

#55

Post by adolph »

"Adversarial" communication with a CV model inference process isn't necessarily an attack because it is unintended by the humans associated with the process. It is more akin to using the full range of an API that uses radiation instead of a network port. It could be used to stage a protest by stopping or slowing cars on a freeway or call attention to deteriorating infrastructure by inducing the car to go over potholes instead of avoiding them. Maybe a neighborhood could self-implement traffic calming measures that don't apply to emergency vehicles.

Re: Adversarial image attacks are no joke

#56

Post by inetknght »

marcosdumay wrote: post #31
mgraczyk wrote: post #14

Earlier quoted context omitted.

I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace " ".

I'm pretty sure this would fail to kill people on almost every place you could try it. And if it works somewhere, it's because there are other problems with the road that should be fixed. Human driving is full of redundancies, and there is a clear hierarchy of information. People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say. If your automated driving system doesn…

> People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say.

You might want to watch the one-way roads in big cities. It happens a lot more often than you assume.

It also is (usually) self-correcting: oncoming traffic will honk, stop, or move around. The offender will (usually) realize their mistake and try to correct.

Sometimes, though, that's not enough. Searching "killed in wrong way one way" on DDG (or assumably Google) yields many (!) news stories.

Re: Adversarial image attacks are no joke

#57

Post by voldacar »

adolph wrote: post #55

"Adversarial" communication with a CV model inference process isn't necessarily an attack because it is unintended by the humans associated with the process. It is more akin to using the full range of an API that uses radiation instead of a network port. It could be used to stage a protest by stopping or slowing cars on a freeway or call attention to deteriorating infrastructure by inducing the car to go over pothole…

If you are trying to make my car go over potholes without my consent, or in any way do something that I don't want it to, that is adversarial behavior. You are my adversary.

Re: Adversarial image attacks are no joke

#58

Post by mgraczyk »

aneutron wrote: post #54
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the c…

Right, and the protections for poison are also security theater for the same reason. In the real world that's ok.

> The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain

It's a quantitative question, just like with computer vision. If you don't like the poison example, consider viral DNA, which is also dangerous in the right hands and does not require massive supply chain control. Not everyone has access to a driving dataset like Teslas, and it would be difficult to trick a Tesla without such a dataset.

We should allow researches to develop attacks, just like we should allow researchers to study poisons, DNA, and viruses.

Re: Adversarial image attacks are no joke

#59

Post by mgraczyk »

ReleaseCandidat wrote: post #49
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

> We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. Actually no. We know that only some psychopaths would do that and so the risk is minimal. AI is currently simply not 'good enough' to be used in critical environments. The problem is that _any_ sticker or even dirt or snow or ... on any road sign can lead to misinterpretation, you can…

Sometime iceberg lettuce kills people (salmonella). We have safety regulations and inspections to mitigate that, but you can never prove that iceberg lettuce is safe.
Post reply on HN