Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adversarial image attacks are no joke

unite.ai

Post reply on HN
Print view View original

11–20 of 196 posts

Re: Adversarial image attacks are no joke

#11

Post by owlbite »

laura_g wrote: post #6

The literature has pretty consistently shown that adversarial examples can be found with only black box access (even with truncated prediction vectors), robustness methods are primarily a cat-and-mouse game between attackers and defenders, and the existence of adversarial examples is likely inevitable ( https://arxiv.org/pdf/1809.02104.pdf ). The big question that remains is - so what? There's exceedingly few use cas…

Self driving cars seem like a dangerous threat vector if an adversarial image can be deployed in such a way as to cause them to commit dangerous maneuvers on demand.

Re: Adversarial image attacks are no joke

#12

Post by pixelgeek »

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

> For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road.

If your vision system can be caused to swerve off a road by a sticker then maybe it shouldn't be used?

Re: Adversarial image attacks are no joke

#13

Post by mgraczyk »

voakbasda wrote: post #9
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

Your proposed laws do not cut out any exemption for research and experimentation, either with existing systems or potential new ones. This level of regulation would create an impossibly high barrier to entry and ensure that only the established players would remain in the marketplace. The last thing that I want to see is yet more regulatory capture, particularly in an industry that has yet to establish a reasonable b…

None of the things I listed would affect research. Researchers shouldn't be posting these on public highways, and researchers shouldn't be distributing them with the intent to cause harm.

Re: Adversarial image attacks are no joke

#14

Post by mgraczyk »

pixelgeek wrote: post #12
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

> For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. If your vision system can be caused to swerve off a road by a sticker then maybe it shouldn't be used?

I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace "".

Re: Adversarial image attacks are no joke

#15

Post by SavantIdiot »

There is a fundamental disconnect between what deep vision models can do and what is expected of them. On the one hand, there is a very good reason why mean-average-precision is used to assess detection-classification models: because even people make mistakes. On the other hand, we need to apply the use of these forever imperfect models with care, context, and redundancy. This is why engineers add a dozen other input types to ADAS systems in addition to vision (sonar, lidar, mesh computing, etc). This is why regulation is needed, to prevent less rigorous products from making their way into situations where the can be easily compromised, or worse, deadly.

Re: Adversarial image attacks are no joke

#16

Post by pixelgeek »

ravi-delia wrote: post #8

It seems to me theres a difference between attacks that carefully craft an image that slips through the cracks, and an attack that basically exploits the fact that without context, it's hard to figure out what single item is important. If I took a picture of a conch shell on top of my keyboard and sent it to someone, no one would think I was just showing off my keyboard! They'd assume, correctly, that my desk was a m…

I don't think that the word 'train' should be used for these systems. We feed then reams of data and effectively cull the ones that don't work but the critical problem is that we judge the effectiveness of an ML system and we actually do know what the ML systems is supposed to be looking for.

We feed a system a series of images of bikes and then select the ones that can pick out a bike but we don't know how the bike is being chosen. We know it is picking out bikes but we have no way to predict if the system is picking out bikes or picking out a series of contrasting colour and shadow shapes and could easily be thrown off by anything that contains the same sort of data.

Re: Adversarial image attacks are no joke

#18

Post by pixelgeek »

mgraczyk wrote: post #14
pixelgeek wrote: post #12

Earlier quoted context omitted.

> For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. If your vision system can be caused to swerve off a road by a sticker then maybe it shouldn't be used?

I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace " ".

But that is not a vision issue. That is providing people with incorrect information.

Re: Adversarial image attacks are no joke

#20

Post by laura_g »

owlbite wrote: post #11
laura_g wrote: post #6

The literature has pretty consistently shown that adversarial examples can be found with only black box access (even with truncated prediction vectors), robustness methods are primarily a cat-and-mouse game between attackers and defenders, and the existence of adversarial examples is likely inevitable ( https://arxiv.org/pdf/1809.02104.pdf ). The big question that remains is - so what? There's exceedingly few use cas…

Self driving cars seem like a dangerous threat vector if an adversarial image can be deployed in such a way as to cause them to commit dangerous maneuvers on demand.

I completely agree, but that's a very big "if". I'm not terribly familiar with autonomous vehicle driving systems, but my passing understanding is that there are multiple components working together that help make predictions, and these systems do not rely on any single point of failure.

The classic example of a sticker on a stop sign is, in my view, more of a dramatization than a real threat surface. Designing an adversarial perturbation on a sticker that can cause misclassifications from particular angles and lighting conditions is possible, but that alone won't cause a vehicle to ignore traffic situations, pedestrians, and other contextual information.

Plus, if I wanted to trick a self driving vehicle into not stopping at an intersection, it would be much easier and cheaper for me to just take the stop sign down :)

Post reply on HN