Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

31–40 of 148 posts

Re: How Adversarial Attacks Work

#31

Post by obastani »

joe_the_user wrote: post #5

"Recent studies by Google Brain have shown that any machine learning classifier can be tricked to give incorrect predictions" -- That has to be an overly broad statement (edit: it might true if you say "neural nets" or something specific instead). I would assume they mean any standard deep learning system and maybe any system that is more or less "generalized regression" but that couldn't be "any machine learning sys…

I think they might be referring [1], which shows that adversarial examples transfer to models such as decision trees and SVMs.

[1] https://arxiv.org/abs/1605.07277

Re: How Adversarial Attacks Work

#32

Post by zardo »

neptvn wrote: post #21

> Apart from the fact that nobody wants to risk having false positives, there’s a simple argument as old as machine learning itself: whatever a human can do, a machine can be taught to do. Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically. So by all means, the authors (and a vast majority of researchers) seem to be confident that ML/DL is the road to AGI…

>Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically.

Not true, we just have different priors and are fooled in different ways. See stage magic, optical illusions, etc...

Re: How Adversarial Attacks Work

#33

Post by yters »

schoen wrote: post #18
munificent wrote: post #17

Earlier quoted context omitted.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

Yes, when I read "Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically" in the article I thought "aren't optical illusions adversarial examples for human perception?" and also "maybe we just haven't found the other kinds of adversarial examples that work against human perception because we can't show humans sufficiently large numbers of examples and measure…

Optical illusions are reverse adversarial examples to those who think the human mind is a deep neural network.

Re: How Adversarial Attacks Work

#34

Post by ChuckMcM »

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street.

Clearly there is a marketing opportunity for t-shirts that make you recognize as other things. Who doesn't want to show up in an image search for toasters on Google Images ? :-)

But my current best guess on how this issue will be addressed will be with classifier diversity an voting systems. While that just moves the problem into a harder and harder to synthesize data set (something that not only is adversarial to one classifier but gives the same answer on several), I believe it will get us to the point where we can trust the level of work to defeat them is sufficiently hard to make it a non-threat.

Re: How Adversarial Attacks Work

#35

Post by matt4077 »

lazugod wrote: post #25

Adversarial attacks sound useful. How can we make it illegal to prevent them?

I may not completely understand the question, but I'm pretty sure "Intentionally fooling an algorithm for nefarious purposes is illegal" is both the answer, and the status quo.

Re: How Adversarial Attacks Work

#36

Post by yters »

danharaj wrote: post #22
munificent wrote: post #17

Earlier quoted context omitted.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

Cool illusion; I figured it out and so will most humans. I don't think the claim that these sorts of illusions are of the same nature as, say, single pixel attacks against neural networks is justified. On the other hand, it is also true that it's not clear whether such attacks on our current crop of primitive neural networks will work on more advanced autonomous systems that regulate themselves.

In theory it seems that with sufficient understanding of any ML system, regardless of sophistication, such examples can be constructed. It is essentially the brain in the vat problem. Computers have no access to objective reality, whereas we cannot say the same regarding the human mind without making strong assumptions about the nature of the human mind.

To put this in comp sci language, we can think of DNNs as proofs that certain inputs belong to certain classes. Since no proof system can be both complete and consistent, something outside the proof system can always either provide unprovable examples or adversarial examples.

Re: How Adversarial Attacks Work

#37

Post by sandGorgon »

Can such methods be used to "fingerprint" proprietary datasets by tainting them ? For example, i want to make sure that my dataset is not stolen and used by someone else (Waymo?). So I taint it using an adversarial method and create a "canary test set" that will uniquely identify if my dataset has been used in some training.

Re: How Adversarial Attacks Work

#38

Post by scrooched_moose »

anishathalye wrote: post #30
scrooched_moose wrote: post #28

Is there any reason to think this would work at all in the real world? All of these "attacks" require complete control of the image being fed to the classifier. In the ATM example you don't directly load an image of the check to the computer inside the machine. You design a check in photoshop, add the noise, print it out, feed it to the machine, which takes a picture of the check. Mobile bank apps still require you t…

Goodfellow et al. demonstrate that adversarial misclassification works in the physical world: https://arxiv.org/abs/1607.02533 -- a printed image is consistently misclassified (demo here: https://www.youtube.com/watch?v=zQ_uMenoBCk ) Recent work by me and some friends demonstrates that physical-world adversarial examples can actually be made quite robust, and you can synthesize 3D adversarial objects as well, and mak…

Thanks for the links and great work! I hadn't seen any research on it making the jump to the real world yet.

Re: How Adversarial Attacks Work

#39

Post by xapata »

alexbeloi wrote: post #19
joe_the_user wrote: post #5

"Recent studies by Google Brain have shown that any machine learning classifier can be tricked to give incorrect predictions" -- That has to be an overly broad statement (edit: it might true if you say "neural nets" or something specific instead). I would assume they mean any standard deep learning system and maybe any system that is more or less "generalized regression" but that couldn't be "any machine learning sys…

>I mean one could imagine a deterministic model that couldn't be "tricked". The models in reference are deterministic, most models for classification/regression are deterministic. Adversarial attacks are showing that the models are chaotic (in the dynamical systems sense), they're very sensitive to their inputs. edit: Not all models have this issue, it's been shown that all the typical image based convolutional neura…

We've known for decades that NNs are prone to overfit. This is just another example of it. With great variance comes great ... uh ... susceptibility to unusual future inputs?

Re: How Adversarial Attacks Work

#40

Post by mikepurvis »

anishathalye wrote: post #30
scrooched_moose wrote: post #28

Is there any reason to think this would work at all in the real world? All of these "attacks" require complete control of the image being fed to the classifier. In the ATM example you don't directly load an image of the check to the computer inside the machine. You design a check in photoshop, add the noise, print it out, feed it to the machine, which takes a picture of the check. Mobile bank apps still require you t…

Goodfellow et al. demonstrate that adversarial misclassification works in the physical world: https://arxiv.org/abs/1607.02533 -- a printed image is consistently misclassified (demo here: https://www.youtube.com/watch?v=zQ_uMenoBCk ) Recent work by me and some friends demonstrates that physical-world adversarial examples can actually be made quite robust, and you can synthesize 3D adversarial objects as well, and mak…

I wouldn't have believed but for those videos. Fascinating stuff.
Post reply on HN