Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

51–60 of 148 posts

Re: How Adversarial Attacks Work

#51

Post by not_that_noob »

ronald_raygun wrote: post #42
scrooched_moose wrote: post #28

Is there any reason to think this would work at all in the real world? All of these "attacks" require complete control of the image being fed to the classifier. In the ATM example you don't directly load an image of the check to the computer inside the machine. You design a check in photoshop, add the noise, print it out, feed it to the machine, which takes a picture of the check. Mobile bank apps still require you t…

Another example where they 3d printed a turtle to fool classifiers https://twitter.com/daniel_bilar/status/925541876484231168

Slick! 84% success rate in the real world, and a simple clever technique. Basically use another DNN to reverse-engineer the target, find weaknesses using the substitute and then use those examples to make attack vectors. Without a robust mathematical framework to understand why a DNN behaves as it does, this is almost impossible to guard against.

From the abstract: "Machine learning (ML) models, e.g., deep neural networks (DNNs), are vulnerable to adversarial examples: malicious inputs modified to yield erroneous model outputs, while appearing unmodified to human observers. Potential attacks include having malicious content like malware identified as legitimate or controlling vehicle behavior. Yet, all existing adversarial example attacks require knowledge of either the model internals or its training data. We introduce the first practical demonstration of an attacker controlling a remotely hosted DNN with no such knowledge. Indeed, the only capability of our black-box adversary is to observe labels given by the DNN to chosen inputs. Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN. We use the local substitute to craft adversarial examples, and find that they are misclassified by the targeted DNN. To perform a real-world and properly-blinded evaluation, we attack a DNN hosted by MetaMind, an online deep learning API. We find that their DNN misclassifies 84.24% of the adversarial examples crafted with our substitute. We demonstrate the general applicability of our strategy to many ML techniques by conducting the same attack against models hosted by Amazon and Google, using logistic regression substitutes. They yield adversarial examples misclassified by Amazon and Google at rates of 96.19% and 88.94%. We also find that this black-box attack strategy is capable of evading defense strategies previously found to make adversarial example crafting harder."

Re: How Adversarial Attacks Work

#52

Post by throwaway613834 »

zardo wrote: post #32
neptvn wrote: post #21

> Apart from the fact that nobody wants to risk having false positives, there’s a simple argument as old as machine learning itself: whatever a human can do, a machine can be taught to do. Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically. So by all means, the authors (and a vast majority of researchers) seem to be confident that ML/DL is the road to AGI…

>Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically. Not true, we just have different priors and are fooled in different ways. See stage magic, optical illusions, etc...

> Not true, we just have different priors and are fooled in different ways. See stage magic, optical illusions, etc...

But we can often recognize them as such, which is important. Actually, on that note, is there work on making machines being able to recognize magic tricks?

Re: How Adversarial Attacks Work

#53

Post by throwaway613834 »

rtkwe wrote: post #46
squeaky-clean wrote: post #14

Earlier quoted context omitted.

> If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). Plausible deniability? "I don't know why it took $1 million for a $100 check, look the check clearly says $100 on it, I didn't edit it. Must be a bank glitch."

All the deposit systems I've used recently still require you to enter the deposit amount so if you're entering 1000000 for a check that 'clearly' says 100 you lose all plausible deniability because you're actively lying to the bank at that point.

Huh, I'd always wondered why they ask you to enter the number. Thanks for pointing this out!

Re: How Adversarial Attacks Work

#54

Post by smogcutter »

projectileboy wrote: post #43
ChuckMcM wrote: post #34

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you…

One possible mitigation could be the use of geocoded roadway metadata. I rented a car in France that had this - it would let me know when I was exceeding the speed limit based on my position. Not hyper-accurate, and so probably not a replacement for all road signs, but certainly good enough for many situations.

Was thinking of this too, but what happens when the map and "ground truth" disagree? The machine would need to be able to identify the rare circumstances when it should trust the map over its senses. If it's capable of that, this wouldn't be a problem in the first place.

Re: How Adversarial Attacks Work

#55

Post by throwaway613834 »

YeGoblynQueenne wrote: post #44
munificent wrote: post #17

Earlier quoted context omitted.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only. It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like tha…

> It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like that.

Sure but people do, for example, mistake each others' faces or voices. You don't need to mistake your friend for a lawnmower for it to be dangerous.

Also, for example, I often mishear my own name when someone else is speaking. Might not happen with every name but it does with some.

Re: How Adversarial Attacks Work

#56

Post by yters »

danharaj wrote: post #47
yters wrote: post #36

Earlier quoted context omitted.

In theory it seems that with sufficient understanding of any ML system, regardless of sophistication, such examples can be constructed. It is essentially the brain in the vat problem. Computers have no access to objective reality, whereas we cannot say the same regarding the human mind without making strong assumptions about the nature of the human mind. To put this in comp sci language, we can think of DNNs as proof…

I think the proper test is how stable the learning system is in the face of adversarial examples. I have no indications either way that we could, using current ideas, create a system that learns and is autonomous like a human brain while regulating itself from going too far off the rails. An optical illusion won't lead to you thinking you're the king of France, nor do there seem to be basilisks that will actually cra…

The incompleteness/inconsistency theorem I mention seems to indicate no such ML system is possible. It will either encounter examples it cannot deal with, or it will be deceived. ML can only be guaranteed to work within very limited domains.

Re: How Adversarial Attacks Work

#57

Post by pdeburen »

While it's true that these attacks work well on state-of-the-art models, there are defence strategies such as including adversial examples during training. Advanced defence strategies such as https://arxiv.org/abs/1705.07204 are robust to a wide array of attacks and achieve very competitive error rates.

I'm not saying it's not a problem but there are successful defence strategies already in place for many attacks.

Re: How Adversarial Attacks Work

#58

Post by munificent »

YeGoblynQueenne wrote: post #44
munificent wrote: post #17

Earlier quoted context omitted.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only. It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like tha…

> It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like that.

Here's a physical object that makes you mistake an insect for a plant:

https://en.wikipedia.org/wiki/Phasmatodea

Re: How Adversarial Attacks Work

#59

Post by munificent »

danharaj wrote: post #47
yters wrote: post #36

Earlier quoted context omitted.

In theory it seems that with sufficient understanding of any ML system, regardless of sophistication, such examples can be constructed. It is essentially the brain in the vat problem. Computers have no access to objective reality, whereas we cannot say the same regarding the human mind without making strong assumptions about the nature of the human mind. To put this in comp sci language, we can think of DNNs as proof…

I think the proper test is how stable the learning system is in the face of adversarial examples. I have no indications either way that we could, using current ideas, create a system that learns and is autonomous like a human brain while regulating itself from going too far off the rails. An optical illusion won't lead to you thinking you're the king of France, nor do there seem to be basilisks that will actually cra…

> An optical illusion won't lead to you thinking you're the king of France

I'm not sure what you're getting at here. By definition, optical illusions are perceptual mistakes limited to the optical system.

It is certainly possible for an optical illusion to trick someone into doing something they don't intend to do because they are fooled into believing the illusion. Imagine painting a set of stairs with a disorienting pattern of shading that makes them appear off, leading you to trip and fall down them.

I don't think it's necessarily for a perceptual illusion to hijack your entire belief system in order to be considered an effective adversarial attack.

Re: How Adversarial Attacks Work

#60

Post by LolWolf »

j2kun wrote: post #10
zitterbewegung wrote: post #8

Earlier quoted context omitted.

No system is 100% secure. What they mean is you can train an adversarial input that would work with a deep learning system and this can fool other machine learning systems. Or that you can create a deep learning system that would fool another (you use a GAN to make forgeries to fool your target ).

> No [machine learning] system is 100% secure. That is what's being claimed without proof, when they meant "existing neural network systems are known to be insecure"

Sure, you can't claim that every machine learning system is susceptible to this attack (here's one that isn't, a model that always returns the same class on all inputs).

But (apart from obvious trivialities) I suspect this is a more general problem with interesting properties—the claim the GP is making boils down to something like: a non-trivial[0] machine learning model with large parameter space is, with vanishing probability (as the number of distinct training samples increases), robust to classes of eps-bounded attacks.[1] This seems like a provable claim that is well-defined and (in a weird, handwavy 'intuitive' kind of way) likely to be true... there are just way too many parameters and too much uncertainty in the local minima that we reach when training from such classes of examples to have 'robustness'.

Perhaps I'm totally wrong, though, and someone will come up with a regularizer that prevents all of this from happening, but this seems highly not obvious to me, at first glance.

---

[0] Non-trivial means that, say, it has non-vanishing curvature (Fisher information, assuming the model returns a vector of probabilities) almost everywhere. I hate to be so nitpicky, but I suspect someone would soon comment asking for definitions of all of these things and that the problem isn't 'well-defined.'

[1] Given a (large enough) set of inputs, the max-norm difference between at least one input and the adversarial example must be ≤eps. In other words, if we want the model to misclassify 'turtle' with 'dog', we don't just give a dog picture---the adversarial picture must look like a turtle in a mathematical sense.

Post reply on HN