Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

11–20 of 148 posts

Re: How Adversarial Attacks Work

#11

Post by matt4077 »

> Lately, safety concerns about AI were revolving around ethics — today we are going to talk about more pressuring[sic] and real issues.

Nice how this casually demeans peoples' worries. Of course, the (demonstrated) idea that ML algorithm would pick up on, and amplify, discrimination (among other things) isn't "real" to these guys.

Risk-scoring of loan applications actually happens to be one of the few uses of ML in the non-tec sector, and it's incredibly likely that some of them are already denying people's application because they happen to be named "La David" and not "Emil". But maybe the authors just don't consider "ethics" to ever be a "real" problem?

But "change a single pixel and the ATM gives you $1,000,000" is, apparently, a "real" and "pressing" problem.

Re: How Adversarial Attacks Work

#12

Post by j2kun »

yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

Take care, the original authors were being overly broad when they implied "all ML algorithms" can be fooled in this way.

Not that I disagree with your conclusion. Just the hypothesis.

Re: How Adversarial Attacks Work

#13

Post by maltalex »

yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

> this shows the human mind is not an ML algorithm.

not necessarily.

Humans have a lot more knowledge about contextual information that today's ML often misses. For every "image" you classify you use a LOT of side data. That side data has also been acquired through learning.

Even if your actually saw a 1,000,000$ check or a 200 km/h speed limit, you'd know it's probably nonsense. Based on your life experience so far, you know that there's no way that mom just gave you a 1,000,000$ check, or that the city decided to turn that small residential street into a racing track. An image classification ML algorithm doesn't know any of that.

Re: How Adversarial Attacks Work

#14

Post by squeaky-clean »

dontreact wrote: post #9

I don’t buy any of the attacks listed here or see how the examples being imperceptible is actually a factor. If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). If you are going to go out and replace speed limit signs to fool self driving cars, it’s probably equally dangerous whether or not the change is obvious, because if it’s way out of bo…

> If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human).

Plausible deniability? "I don't know why it took $1 million for a $100 check, look the check clearly says $100 on it, I didn't edit it. Must be a bank glitch."

Re: How Adversarial Attacks Work

#15

Post by bertil »

joe_the_user wrote: post #5

"Recent studies by Google Brain have shown that any machine learning classifier can be tricked to give incorrect predictions" -- That has to be an overly broad statement (edit: it might true if you say "neural nets" or something specific instead). I would assume they mean any standard deep learning system and maybe any system that is more or less "generalized regression" but that couldn't be "any machine learning sys…

It could be this: http://www.labsix.org/physical-objects-that-fool-neural-nets...

It is impressive, but that came out a couple of days ago and is not reviewed yet, as far as I can tell.

It’s not obvious from the URL but two of the authors were interns at Google, which could explain why they describe some aspects of InceptionV3 as white-box (because it's not clear if they have re-trained it themselves).

Re: How Adversarial Attacks Work

#16

Post by maltalex »

dontreact wrote: post #9

I don’t buy any of the attacks listed here or see how the examples being imperceptible is actually a factor. If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). If you are going to go out and replace speed limit signs to fool self driving cars, it’s probably equally dangerous whether or not the change is obvious, because if it’s way out of bo…

> If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human).

Humans are harder to fool. But some bank apps allow you to deposit a check by photographing it. Such apps would be fairly easy to attack.

If you limit yourself to making 100$ checks become 1000$ checks and not 1,000,000$ checks, you might even get away with it.

Re: How Adversarial Attacks Work

#17

Post by munificent »

yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

I think your logic is:

1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm.

But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one:

http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle...

This is exactly what an optical illusion is.

Re: How Adversarial Attacks Work

#18

Post by schoen »

munificent wrote: post #17
yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

Yes, when I read "Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically" in the article I thought "aren't optical illusions adversarial examples for human perception?" and also "maybe we just haven't found the other kinds of adversarial examples that work against human perception because we can't show humans sufficiently large numbers of examples and measure the humans' responses accurately enough".

But the optical illusion example seems very strong because if you say that human perception includes abilities to classify things, there are many examples just like your link that show cases where the classification routinely goes wrong.

It seems possible to me that there are as-yet undiscovered adversarial examples in human perception that we simply don't have any feasible way to search for, and may never be able to construct practical examples of. (It would probably be extremely unnerving to experience one in real life.)

Re: How Adversarial Attacks Work

#19

Post by alexbeloi »

joe_the_user wrote: post #5

"Recent studies by Google Brain have shown that any machine learning classifier can be tricked to give incorrect predictions" -- That has to be an overly broad statement (edit: it might true if you say "neural nets" or something specific instead). I would assume they mean any standard deep learning system and maybe any system that is more or less "generalized regression" but that couldn't be "any machine learning sys…

>I mean one could imagine a deterministic model that couldn't be "tricked".

The models in reference are deterministic, most models for classification/regression are deterministic.

Adversarial attacks are showing that the models are chaotic (in the dynamical systems sense), they're very sensitive to their inputs.

edit: Not all models have this issue, it's been shown that all the typical image based convolutional neural networks are susceptible to this issue. My guess is that it's a more general problem of high dimensional inputs.

Re: How Adversarial Attacks Work

#20

Post by joe_the_user »

zitterbewegung wrote: post #8
joe_the_user wrote: post #5

"Recent studies by Google Brain have shown that any machine learning classifier can be tricked to give incorrect predictions" -- That has to be an overly broad statement (edit: it might true if you say "neural nets" or something specific instead). I would assume they mean any standard deep learning system and maybe any system that is more or less "generalized regression" but that couldn't be "any machine learning sys…

No system is 100% secure. What they mean is you can train an adversarial input that would work with a deep learning system and this can fool other machine learning systems. Or that you can create a deep learning system that would fool another (you use a GAN to make forgeries to fool your target ).

"Machine learning" is a broader category than "deep learning", just saying.
Post reply on HN