Adversarial image attacks are no joke
1–10 of 196 posts
Re: Adversarial image attacks are no joke
#2Language models have the same defect - they are quite brittle and susceptible to black-box adversarial attacks ( eg arxiv.org/abs/1907.11932 )
Re: Adversarial image attacks are no joke
#3This is bad news for safety critical computer vision systems like Tesla vision.
Re: Adversarial image attacks are no joke
#4The really scary thing is that this could be used as an excuse to hide production ML models and even the tech used to generate them. Sounds like we can expect the state-of-the-art AI techniques to be jealously guarded eventually. I guess optimism on the ground is enough to have prevented that so far, but once the scales tip away from sharing and towards exploitation.. well, we know it's largely a one-way process on the 1 decade time scale. Is this the chilling effect that will bring us into the next AI winter?
Re: Adversarial image attacks are no joke
#5The really scary thing is that this could be used as an excuse to hide production ML models and even the tech used to generate them. Sounds like we can expect the state-of-the-art AI techniques to be jealously guarded eventually. I guess optimism on the ground is enough to have prevented that so far, but once the scales tip away from sharing and towards exploitation.. well, we know it's largely a one-way process on t…
> Sounds like we can expect the state-of-the-art AI techniques to be jealously guarded eventually.
This isn’t an eventuality, it’s the current state of the industry.
Re: Adversarial image attacks are no joke
#6The literature has pretty consistently shown that adversarial examples can be found with only black box access (even with truncated prediction vectors), robustness methods are primarily a cat-and-mouse game between attackers and defenders, and the existence of adversarial examples is likely inevitable (https://arxiv.org/pdf/1809.02104.pdf).
The big question that remains is - so what? There's exceedingly few use cases where the existence of adversarial examples causes a security threat. There's a lot of research value in understanding adversarial examples and what that tells us about how models learn, generalize, and retain information, but I am not convinced that these attacks pose a threat remotely close to the amount of attention given.
Re: Adversarial image attacks are no joke
#7As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison.
It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though.
It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Instead, we rely on weaker protections. We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison.
I believe the same will be true of adversarial examples for vision (and language) models. We can try to make them hard to make, hard to posses anonymously, and hard to deliver. I think this will be much easier with computer vision than with poison, so I'm not worried about it.
For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. Governments should protect people from this in multiple ways, similarly to how they protect us from poison:
1. People who post these stickers should go to prison.
2. People who create and distribute these stickers knowing their purpose should go to prison.
3. Tesla should be civilly liable for cases where preventing such an incident was possible with known technology.
4. Roads should be modified over time to make it more difficult to do this attack.
Re: Adversarial image attacks are no joke
#8It seems to me theres a difference between attacks that carefully craft an image that slips through the cracks, and an attack that basically exploits the fact that without context, it's hard to figure out what single item is important. If I took a picture of a conch shell on top of my keyboard and sent it to someone, no one would think I was just showing off my keyboard! They'd assume, correctly, that my desk was a mess and I didn't feel like finding a clear surface.
That's not to say that either attack is less harmful than the other! If you train an image classifier to find bikers, it's not really wrong or right to say that a picture of a biker qualifies. But if a car stops lest it run over a painted bike on the road, that's obviously bad. The problem is that you aren't trying to recognize bikers, you're trying to avoid obstacles. We just don't train well for that.
Re: Adversarial image attacks are no joke
#9As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…
Your proposed laws do not cut out any exemption for research and experimentation, either with existing systems or potential new ones. This level of regulation would create an impossibly high barrier to entry and ensure that only the established players would remain in the marketplace. The last thing that I want to see is yet more regulatory capture, particularly in an industry that has yet to establish a reasonable baseline of success.
Re: Adversarial image attacks are no joke
#10Maybe this is a good example of why ML systems shouldn't be used? Ultimately we don't know how the networks that get created actually make decisions so doesn't that make protecting them from attacks like this impossible?