Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adversarial image attacks are no joke

unite.ai

Post reply on HN
Print view View original

41–50 of 196 posts

Re: Adversarial image attacks are no joke

#41

Post by bryanrasmussen »

marcosdumay wrote: post #31
mgraczyk wrote: post #14

Earlier quoted context omitted.

I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace " ".

I'm pretty sure this would fail to kill people on almost every place you could try it. And if it works somewhere, it's because there are other problems with the road that should be fixed. Human driving is full of redundancies, and there is a clear hierarchy of information. People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say. If your automated driving system doesn…

the claim is not that automated driving systems are ready for use, the claim is that if you do things in order to compromise a system that has a good chance of killing people and then does kill people that should be illegal, which of course it already is.

Re: Adversarial image attacks are no joke

#43

Post by laura_g »

OldHand2018 wrote: post #40

I see a completely different attack vector here. Lawyers. If you are selling a product or service that has been trained on a dataset that contains copyrighted photos you don't have permission to use and I can "prove it" enough to get you into court and into the discovery phase, you are screwed. I'll get an injunction that shuts you down while we talk about how much money you have to pay me. And lol, if any of those p…

This would be membership inference attacks - https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7958568...

Re: Adversarial image attacks are no joke

#44

Post by goatlover »

dontreact wrote: post #24

If you really wanted to crash cars by altering their visual input, why would you bother with all this complexity? Why not just actually swap the road sign? Why does the existence of these attacks change the threat landscape at all? If people are already not doing "dumb" attacks like just changing/removing road signs why would they start doing them? The risk of messing with road signs and throwing off autonomous vehic…

> Why not just actually swap the road sign?

Because you have to physically do it, as opposed to hacking from anywhere else on the planet.

> not on a system that has a more general sense of collision-avoidance and situational awareness (like humans do).

Are vision systems to that point yet when it comes to driving vehicles?

> Because the whole point of using an automated vision system is usually that you want to avoid human eyes on the problem.

And the point of hacking an automated system is that it's easier to do that remotely than to cause a human to crash locally.

Re: Adversarial image attacks are no joke

#45

Post by b3morales »

pixelgeek wrote: post #18
mgraczyk wrote: post #14

Earlier quoted context omitted.

I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace " ".

But that is not a vision issue. That is providing people with incorrect information.

The specific trick doesn't really matter; the point is that it's possible to maliciously create a situation that makes human pilots act dangerously. We accept that the possibility can't be made nil, and we have post facto rules to deal with it. The same principle applies to traps for machines.

Re: Adversarial image attacks are no joke

#46

Post by gmadsen »

jstanley wrote: post #35
mgraczyk wrote: post #14

Earlier quoted context omitted.

I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace " ".

I think in almost all cases that would not cause a crash. The drivers would see the oncoming traffic and stop rather than crash.

that assumes you can see the threat, if instead it led to an unprotected crossing at high speed, then you have a very different situation

Re: Adversarial image attacks are no joke

#47

Post by OldHand2018 »

laura_g wrote: post #43
OldHand2018 wrote: post #40

I see a completely different attack vector here. Lawyers. If you are selling a product or service that has been trained on a dataset that contains copyrighted photos you don't have permission to use and I can "prove it" enough to get you into court and into the discovery phase, you are screwed. I'll get an injunction that shuts you down while we talk about how much money you have to pay me. And lol, if any of those p…

This would be membership inference attacks - https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7958568...

Oh excellent. But of course the key addition is handing off this information to lawyers who use it to shut you down and/or extract money from you.

If you are using some torrent of a dataset, nobody is indemnifying you, and once you get to the discovery phase of a lawsuit, they are going to know that you intentionally grabbed a dataset you knew you shouldn't have had access to. Treble damages!

Re: Adversarial image attacks are no joke

#48

Post by Laremere »

voakbasda wrote: post #9
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

Your proposed laws do not cut out any exemption for research and experimentation, either with existing systems or potential new ones. This level of regulation would create an impossibly high barrier to entry and ensure that only the established players would remain in the marketplace. The last thing that I want to see is yet more regulatory capture, particularly in an industry that has yet to establish a reasonable b…

INAL, but actually putting adversarial image attacks on real roads is already illegal. If you modify a street sign, and as a result someone dies, that's a fairly easy case of Involuntary manslaughter.

At a minimum, you can't modify street signs. Eg in Washington State:

  RCW 47.36.130
  Meddling with signs prohibited.
  No person shall without lawful authority attempt to or in fact _alter_, deface, injure, knock down, or remove any official traffic control signal, _traffic device_ or railroad sign or signal, or any inscription, shield, or insignia thereon, or any other part thereof.
(Underscore emphasis added). And if you're thinking about not putting it on a sign, but putting it elsewhere visible to cars:

  RCW 46.61.075.1
  Display of unauthorized signs, signals, or markings.
  No person shall place, maintain or display upon or in view of any highway any unauthorized sign, signal, _marking or device_ which purports to be or is an imitation of or resembles an official traffic-control device or railroad sign or signal, or _which attempts to direct the movement of traffic_, or which hides from view or interferes with the effectiveness of an official traffic-control device or any railroad sign or signal.
Where I'm unsure is producing these with the intent or knowledge that they will/could be used by someone to go do this. None of this makes using these for research and experimentation illegal.

Re: Adversarial image attacks are no joke

#49

Post by ReleaseCandidat »

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

> We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison.

Actually no. We know that only some psychopaths would do that and so the risk is minimal.

AI is currently simply not 'good enough' to be used in critical environments. The problem is that _any_ sticker or even dirt or snow or ... on any road sign can lead to misinterpretation, you can never proof that it's safe.

Re: Adversarial image attacks are no joke

#50

Post by pueblito »

siboehm wrote: post #23
mgraczyk wrote: post #14

Earlier quoted context omitted.

I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace " ".

This has always confused me as well. What would be the reason why some adversary would choose to craft an adversarial example and deploy it in the real world versus the much easier solution to just remove / obscure the sign?

Warfare comes to mind, as weapons gain increasingly powerful ai functions and become autonomous
Post reply on HN