Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

81–90 of 148 posts

Re: How Adversarial Attacks Work

#81

Post by ChuckMcM »

Houshalter wrote: post #80
ChuckMcM wrote: post #34

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you…

I mean people can already do that to human drivers. You can easily remove a stop sign. Or put a sticker over an important sign. Any human driver will be fooled! Yet this doesn't seem to be a prevalent problem.

And they have: http://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-y...

It just makes it harder to catch who did it.

Re: How Adversarial Attacks Work

#82

Post by brudgers »

ChuckMcM wrote: post #34

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you…

Based on the history of artificial intelligence, I am less optimistic about the long term prospects of any silver bullet technology, and that's how neural networks and related systems are currently positioned. Ultimately, rule based systems are rule based systems, our inability to articulate the explicit rules should not be mistaken for their absence. Ten or one hundred or a thousand rule based classifiers voting is more of the same, not something different. The black box still has the same inputs and the same outputs no matter how many times the handle of the Pumping Lemma is raised and lowered.

Re: How Adversarial Attacks Work

#83

Post by dontreact »

maltalex wrote: post #16
dontreact wrote: post #9

I don’t buy any of the attacks listed here or see how the examples being imperceptible is actually a factor. If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). If you are going to go out and replace speed limit signs to fool self driving cars, it’s probably equally dangerous whether or not the change is obvious, because if it’s way out of bo…

> If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). Humans are harder to fool. But some bank apps allow you to deposit a check by photographing it. Such apps would be fairly easy to attack. If you limit yourself to making 100$ checks become 1000$ checks and not 1,000,000$ checks, you might even get away with it.

You have to type in a number. So if you are going to type in 1000, it would probably more likely be a successful attack if you actually forged the check so it looked like it said 1000, so again not seeing how an adversarial example is helpful here.

Re: How Adversarial Attacks Work

#84

Post by haraldurt »

sandGorgon wrote: post #37

Can such methods be used to "fingerprint" proprietary datasets by tainting them ? For example, i want to make sure that my dataset is not stolen and used by someone else (Waymo?). So I taint it using an adversarial method and create a "canary test set" that will uniquely identify if my dataset has been used in some training.

In one sense, no. You can guarantee privacy of any given input (or any subset of k inputs) by applying transfer learning of an ensemble of models trained on subsets of the training data [0][1]. This is useful if, for instance, you train on medical data and you don't want anyone to know that "John Doe, HIV+" was part of the input. If your adversary does not take such precautions, however, then your canary should work.

[0] https://arxiv.org/abs/1610.05755

[1] https://static.googleusercontent.com/media/research.google.c...

Re: How Adversarial Attacks Work

#85

Post by Houshalter »

ChuckMcM wrote: post #81
Houshalter wrote: post #80

Earlier quoted context omitted.

I mean people can already do that to human drivers. You can easily remove a stop sign. Or put a sticker over an important sign. Any human driver will be fooled! Yet this doesn't seem to be a prevalent problem.

And they have: http://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-y... It just makes it harder to catch who did it.

It's also harder to do. It requires technical knowledge and some planning, not some drunk teenagers.

Re: How Adversarial Attacks Work

#86

Post by haraldurt »

naveen99 wrote: post #71
Winterflow3r wrote: post #49

Earlier quoted context omitted.

No, this paper by Papernot et al shows how to do blackbox attacks without knowledge of model internals. https://arxiv.org/abs/1602.02697

Thanks. But they still need to be able to use the black box a brute force number of times on shady inputs and get detailed outputs during their gradient descent. Not going to be allowed on a check reader or anything sensitive.

Not necessarily. Adversarial examples have been shown to, for instance, be transferable across different networks with different hyperparameters (e.g., number of layers) trained on disjoint subsets of a training set [0, section 4.2]. There are more references from the paper linked by the OP.

[0] https://arxiv.org/abs/1312.6199

Re: How Adversarial Attacks Work

#87

Post by shmel »

j2kun wrote: post #10
zitterbewegung wrote: post #8

Earlier quoted context omitted.

No system is 100% secure. What they mean is you can train an adversarial input that would work with a deep learning system and this can fool other machine learning systems. Or that you can create a deep learning system that would fool another (you use a GAN to make forgeries to fool your target ).

> No [machine learning] system is 100% secure. That is what's being claimed without proof, when they meant "existing neural network systems are known to be insecure"

Really? I believe they meant exactly what they said. Read https://arxiv.org/abs/1605.07277

Adversarial examples can be easily crafted for linear regression, SVM, decision trees and k-NN at very least. I can't think about any ML technique that is proven to be secure against them.

There is also an old blogpost from Karpathy describing that the only reason we are not concerned about adversarial examples in linear models is that nobody uses them to classify ImageNet.

Re: How Adversarial Attacks Work

#88

Post by empath75 »

munificent wrote: post #17
yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

I know fmri isn't high enough resolution for this, but I wonder if you had real time access to brain state, you could construct adversarial examples like these.

Re: How Adversarial Attacks Work

#90

Post by shmel »

d--b wrote: post #69

Research is needed in that area, but surely adding some noise / blurring to sample during training and during recognition should help reduce the feasibility of the attack. Isn't that the case? Adversarial attacks seem to point to overtraining in some sense.

Unfortunately, it is not that simple. There are multiple ways to generate adversarial examples and simple defenses help at most against the simplest attacks. Every now and again a new preprint "one simple trick defeating the latest adversarial defense technique" appears on arxiv.

I believe they are linked to the nature of DL (and ML in general) models. We try to capture very tiny manifold of natural images in the space of all possible images. We found a technique that does it well (CNNs). But by the very definition of its training we train it to output some values in a finite number of points. In the same time CNN's output outside of those points are mainly defined by the model's smoothness. We can expect that in a neighborhood of a given image the output will be roughly the same (it allows them to generalize). Far from any point of training set the model can say anything. Adversarial examples basically hint that this smoothness work well only along natural images manifold, once we step outside it is much more chaotic. Or, equivalently, the neighborhood where the CNN gives roughly the same output is a very thin "slice" that closely follows natural images manifold. Why is that? Probably there are some non-trivial topological reasons. But it is exactly what nobody understands now.

Post reply on HN