Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Robust Adversarial Examples

blog.openai.com

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Robust Adversarial Examples

#11

Post by nullc »

skishore wrote: post #3
therajiv wrote: post #2

It's not clear to me how malicious actors can manipulate this observation to confuse self-driving cars. That said, I don't think this discredits the point of the article; it's important to note how easily deep learning models can be fooled if you understand the math behind them. I just think the example of tricking self-driving cars is difficult to relate with / understand.

Why do you say that? The first demo they provide shows that the adversarial image, when printed and then manipulated, still fools the algorithm. That means that the example is robust to various affine transformations but also to the per-pixel noise that is a result of a printing something and then viewing it again through a camera. Suppose you were to place an example like that on a stop sign that fooled a car into t…

Though I generally agree with your point, the tree vs stopsign example may not be the best because it would arguably work equally well on humans.

Re: Robust Adversarial Examples

#12

Post by netinstructions »

nullc wrote: post #11
skishore wrote: post #3

Earlier quoted context omitted.

Why do you say that? The first demo they provide shows that the adversarial image, when printed and then manipulated, still fools the algorithm. That means that the example is robust to various affine transformations but also to the per-pixel noise that is a result of a printing something and then viewing it again through a camera. Suppose you were to place an example like that on a stop sign that fooled a car into t…

Though I generally agree with your point, the tree vs stopsign example may not be the best because it would arguably work equally well on humans.

Did the perturbed image of the cat in the article look like a desktop computer to you?

The point is that humans would see one thing whereas computers would be highly confident it is something else.

Re: Robust Adversarial Examples

#13

Post by kevinnk »

tachyonbeam wrote: post #9

IMO, what these adversarial examples give us is a way to boost training data. We should augment training datasets with adversarial examples, or use adversarial training methods. The resulting networks would only be more robust as a result. As for self-driving cars, this is a good argument for having multiple sensing modalities in addition to visual, such as radar/lidar/sonar, and multiple cameras, infrared in additio…

> IMO, what these adversarial examples give us is a way to boost training data.

That's basically the idea behind GANs.

Re: Robust Adversarial Examples

#14

Post by andbberger »

tachyonbeam wrote: post #9

IMO, what these adversarial examples give us is a way to boost training data. We should augment training datasets with adversarial examples, or use adversarial training methods. The resulting networks would only be more robust as a result. As for self-driving cars, this is a good argument for having multiple sensing modalities in addition to visual, such as radar/lidar/sonar, and multiple cameras, infrared in additio…

But at what point do you have to wonder if we're using the wrong basis? And how do you know that augmenting the data with tiny adversarial perturbations won't just leave the network vulnerable in a different direction?

It's pretty obvious how to build translational symmetry into a net that's still expressive and easy to train (convolution). But you have to spoon feed CNNs rotational and other symmetries by augmenting the training data. What you really want is a model that has all the symmetries your data has built in.

My sense is that the community at large seems to regard DL as a magic blackbox which it really is not. Complete basis of function + finite data = guarantee of wonky interpolation between samples. What you really need to do is restrict the class of expressible functions to those you need - build your prior into the model.

Re: Robust Adversarial Examples

#15

Post by DonbunEf7 »

tachyonbeam wrote: post #9

IMO, what these adversarial examples give us is a way to boost training data. We should augment training datasets with adversarial examples, or use adversarial training methods. The resulting networks would only be more robust as a result. As for self-driving cars, this is a good argument for having multiple sensing modalities in addition to visual, such as radar/lidar/sonar, and multiple cameras, infrared in additio…

You are, unfortunately, probably just playing out Mr. Crab's obsession with record players.

Remember that these tricky images are based on the principle that machine-learning algorithms are differentiable and high-dimensional. There is a lot of ways to transition between, say, the desktop dimension and the cat dimension, and it's all continuous, so we're guaranteed to be able to influence the machine in that sort of direction.

You could imagine somehow taking all of the adversarial examples and categorically augmenting a machine's learning to know about the examples, creating a cat-masquerading-as-desktop dimension. But all you've done is make a lot more space (by adding a dimension) and so the next iteration of adversarial examples will be able to proceed by the same process as before, just on this new augmented machine.

Re: Robust Adversarial Examples

#18

Post by hyperion2010 »

My own view of this having spent some time in visual neuroscience is that if you really want vision that is robust to these kinds of issues then you have to build a geometric representation of the world first, and then learn/map categories from that. Trying to jump from a matrix to a label without having an intervening topological/geometric model of the world in between (having 2 eyes and/or the ability to move and help with this) is asking for trouble because we think we are recapitulating biology when in fact we are doing nothing of the sort (as these adversarial examples reveal beautifully).

Re: Robust Adversarial Examples

#19

Post by clickok »

std_throwaway wrote: post #17

Does this effect carry over to classifiers which were trained with different training data?

I am unsure what you mean-- do you mean with different training sets but the same testing set?

It's an interesting question; maybe the reason for (some) of these adversarial vulnerabilities is due to a handful of bad training examples. You could formulate it as a search problem to see if there's particular images (or small groups of images) that are responsible for the adversarial vulnerabilities. This might then indicate that some of these perturbations are really just taking advantage of the fact that neural nets tend to "memorize" some of the data, so we're not really exploiting some deep structural feature so much as just feeding the echo of an input that the net has learned to automatically classify as, say, a computer/desk[0].

It would be a good project, but I don't have enough GPUs on hand to train scores of deep nets from scratch.

Assuming one were to bite the bullet, it might also be worth trying different data augmentation strategies. Most of the time, we try to eke out additional performance/robustness by using the same sets of transformations (translation, rotation, cropping, rescaling, etc.), but if the net is vulnerable to adversarial examples because of something in the training set, then you might just be making sure that adversarial vulnerability is present everywhere in the image and at multiple scales.

On a related note, there's an interesting paper about universal adversarial perturbations, i.e. those that can be added to any image and thereby induce a misclassification with high probability[1]. This effect holds even across different models, so the same perturbation can cause a misclassification in different architectures.

------

0. Neural nets learn by some combination of abstraction and memorization. If, for some reason, many members of a particular class are hard to generalize, then it's possible that they instead learn to identify some particular aspects of those classes (that are not usually present in other images) and have a disproportionate response when those features are present. If such features are not obvious to human visual inspection, then we get misclassifications without insight into why they were misclassified.

1. https://arxiv.org/pdf/1610.08401.pdf

Re: Robust Adversarial Examples

#20

Post by IanCal »

hyperion2010 wrote: post #18

My own view of this having spent some time in visual neuroscience is that if you really want vision that is robust to these kinds of issues then you have to build a geometric representation of the world first, and then learn/map categories from that. Trying to jump from a matrix to a label without having an intervening topological/geometric model of the world in between (having 2 eyes and/or the ability to move and h…

Some disjointed thoughts.

> we think we are recapitulating biology when in fact we are doing nothing of the sort (as these adversarial examples reveal beautifully).

I'm not sure I'd go so far. There's a pretty long list of optical illusions. Seeing motion where there clearly is none, not comparing distances correctly and most relevant here is things that look like a face. Here are a few selected famous examples: http://brainden.com/face-illusions.htm

Some of those immediately make my brain flag up "FACE". It's only looking in more detail that I see what else is there, but my visual system is clearly being tricked, as would billions of other completely independently grown visual systems. How much better could we do this, and with more subtlety, if we could analyse the whole brain like we can neural networks and target a specific brain?

There's an old experiment showing how a kitten raised never seeing horizontal lines will fail to see them ever after a certain age, so we know that biological systems struggle with limited visual input.

I'd also say we're doing matrix -> label conversions ourselves, too, unless we're born with a special geometric model. Deep learning also does things in layers, so there's not a direct matrix-label learning happening straight away, that should come much later after the system has learned to create a higher level representation of the input.

On a less contrarian side, I wonder how well these things would work if we were to show the networks videos of... everything. Years and years of video. Don't try and add labels yet, but can we add a constraint that we expect the representation to only change slowly? Two very similar frames should not result in the high-level interpretation changing drastically.

Post reply on HN