Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

61–70 of 148 posts

Re: How Adversarial Attacks Work

#61

Post by LolWolf »

pdeburen wrote: post #57

While it's true that these attacks work well on state-of-the-art models, there are defence strategies such as including adversial examples during training. Advanced defence strategies such as https://arxiv.org/abs/1705.07204 are robust to a wide array of attacks and achieve very competitive error rates. I'm not saying it's not a problem but there are successful defence strategies already in place for many attacks.

> Although our models are more vulnerable to white-box FGSM samples compared to the v3adv model, ensemble adversarial training significantly increases robustness to black-box attacks that transfer FGSM samples crafted on the holdout Inception v4.

So, I just train my new adversary on the "new" model that was trained on the previous adversarial examples. And now we're back to square one.

I suspect the problem of adversarial attacks is a problem of high-dimensional spaces, not of training on particular samples.

Re: How Adversarial Attacks Work

#62

Post by fenwick67 »

ronald_raygun wrote: post #42
scrooched_moose wrote: post #28

Is there any reason to think this would work at all in the real world? All of these "attacks" require complete control of the image being fed to the classifier. In the ATM example you don't directly load an image of the check to the computer inside the machine. You design a check in photoshop, add the noise, print it out, feed it to the machine, which takes a picture of the check. Mobile bank apps still require you t…

Another example where they 3d printed a turtle to fool classifiers https://twitter.com/daniel_bilar/status/925541876484231168

Wow this is incredible

Re: How Adversarial Attacks Work

#64

Post by yters »

munificent wrote: post #59
danharaj wrote: post #47

Earlier quoted context omitted.

I think the proper test is how stable the learning system is in the face of adversarial examples. I have no indications either way that we could, using current ideas, create a system that learns and is autonomous like a human brain while regulating itself from going too far off the rails. An optical illusion won't lead to you thinking you're the king of France, nor do there seem to be basilisks that will actually cra…

> An optical illusion won't lead to you thinking you're the king of France I'm not sure what you're getting at here. By definition, optical illusions are perceptual mistakes limited to the optical system. It is certainly possible for an optical illusion to trick someone into doing something they don't intend to do because they are fooled into believing the illusion. Imagine painting a set of stairs with a disorientin…

The whole point of these examples is the DNN gives erroneous classifications with near complete certainty. This looks to me like hijacking a belief system.

Re: How Adversarial Attacks Work

#65

Post by jpttsn »

YeGoblynQueenne wrote: post #44
munificent wrote: post #17

Earlier quoted context omitted.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only. It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like tha…

https://m.youtube.com/watch?v=A4QcyW-qTUg

Re: How Adversarial Attacks Work

#66

Post by romaniv »

munificent wrote: post #58
YeGoblynQueenne wrote: post #44

Earlier quoted context omitted.

The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only. It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like tha…

> It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like that. Here's a physical object that makes you mistake an insect for a plant: https://en.wikipedia.org/wiki/Phasmatodea

Problem is, mimicry involves copying essential properties of the target object: color, texture, shape, movement dynamics and so on. It relies on true ambiguity. Adversarial examples against neural networks (the interesting ones, anyway) involve a combination of insignificant, seemingly random permutations that only work in their totality. That's a very important difference.

Re: How Adversarial Attacks Work

#67

Post by pdeburen »

LolWolf wrote: post #61
pdeburen wrote: post #57

While it's true that these attacks work well on state-of-the-art models, there are defence strategies such as including adversial examples during training. Advanced defence strategies such as https://arxiv.org/abs/1705.07204 are robust to a wide array of attacks and achieve very competitive error rates. I'm not saying it's not a problem but there are successful defence strategies already in place for many attacks.

> Although our models are more vulnerable to white-box FGSM samples compared to the v3adv model, ensemble adversarial training significantly increases robustness to black-box attacks that transfer FGSM samples crafted on the holdout Inception v4. So, I just train my new adversary on the "new" model that was trained on the previous adversarial examples. And now we're back to square one. I suspect the problem of advers…

I'm not sure how the quote supports your argument. Adversial examples generalize well accross many different classifiers.

Shallow NN's can be fooled just as well, it seems to be more of a problem of linear models in general. Apparently Geoff Hintons Capsule Networks are more robust due to being "less linear" (Ian Goodfellow mentioned this in a recent talk, don't have the references now to back it up)

Re: How Adversarial Attacks Work

#69

Post by d--b »

Research is needed in that area, but surely adding some noise / blurring to sample during training and during recognition should help reduce the feasibility of the attack. Isn't that the case?

Adversarial attacks seem to point to overtraining in some sense.

Re: How Adversarial Attacks Work

#70

Post by romaniv »

munificent wrote: post #17
yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

The comparison in disingenuous. This illusion has nothing to do with image classification and heavily relies on adding misleading context to the object to alter the perception of one of its properties. Most optical illusions are like that. That is not what is being done with adversarial examples.
Post reply on HN