Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Robust Adversarial Examples

blog.openai.com

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Robust Adversarial Examples

#31

Post by yorwba »

DonbunEf7 wrote: post #15
tachyonbeam wrote: post #9

IMO, what these adversarial examples give us is a way to boost training data. We should augment training datasets with adversarial examples, or use adversarial training methods. The resulting networks would only be more robust as a result. As for self-driving cars, this is a good argument for having multiple sensing modalities in addition to visual, such as radar/lidar/sonar, and multiple cameras, infrared in additio…

You are, unfortunately, probably just playing out Mr. Crab's obsession with record players. Remember that these tricky images are based on the principle that machine-learning algorithms are differentiable and high-dimensional . There is a lot of ways to transition between, say, the desktop dimension and the cat dimension, and it's all continuous, so we're guaranteed to be able to influence the machine in that sort of…

But we don't really care about the cat-masquerading-as-desktop category in itself, so an adversarial example that makes a cat look like a cat-masquerading-as-desktop, or masquerades a cat-masquerading-as-desktop as a cat, isn't really relevant.

By adding enough adversarial examples to the training set, you can absolutely immunize a model against adversarial perturbations of the training data.

The problem is that the volume of "not very different" data points surrounding an example grows exponentially with the input dimension, so you need to train for much longer, and your "adversarial protection" will likely overfit to the neighborhood of training examples, which doesn't help with unseen data.

Re: Robust Adversarial Examples

#32

Post by mjn »

clickok wrote: post #21
hyperion2010 wrote: post #18

My own view of this having spent some time in visual neuroscience is that if you really want vision that is robust to these kinds of issues then you have to build a geometric representation of the world first, and then learn/map categories from that. Trying to jump from a matrix to a label without having an intervening topological/geometric model of the world in between (having 2 eyes and/or the ability to move and h…

We tried that; the reason deep nets are popular is that they outperform geometric (or other problem-specific) models. This might be because they implicitly develop such representations somewhere along the way, or because such a representation is not really necessary for visual classification. Additionally, introducing ancillary modules is not without cost-- you might gain robustness to some kinds of adversarial input…

> outperform

I think that remains to be seen, at least in the general case, since we haven't yet agreed on a measure of performance. The debate around adversarial examples can be interpreted as arguing over the proper measure of performance. Although so far the debate is doing so somewhat implicitly, since afaik nobody has formalized a measure of robustness to adversarial examples; it's progressed more by case studies (which is fine, since research into NN robustness is still quite early stage, and case studies can help illustrate issues). I think it can be fairly said that neural nets perform well on the ImageNet benchmark and similar measures of performance. But whether those are good measures of performance, or whether some kind of metric that weights robustness more heavily should be used (and what methods would perform well on that) is the subject of current research, like this research.

Re: Robust Adversarial Examples

#33

Post by yorwba »

mamp wrote: post #22
sagebird wrote: post #10

Earlier quoted context omitted.

It doesn't matter how many algorithms or sensors are consulted or combined to form judgment. If an attacker can obtain a self driving vehicle's hardware, and if enough tests can be performed per seconds, the attacker can train images that fool it. Your idea is similar to an appeal to security through obscurity. Might work sometimes, but not generally. (Noise does not help, because you can still discover a gradient to…

You assume that all sensors are working on 'images' and that the algorithm are all using gradient descent.

Replace 'images' with 'sensor data' and adversarial examples can still be generated. They might not be as easy to feed into the vehicles hardware (e.g. requiring speakers to fool an acoustic sensor), but the same principles apply.

It's also not necessary for the recognition algorithms to be using gradient descent, so long as they are differentiable (or can be approximated by a model that is), you can use gradient descent to find adversarial examples.

Adversarial examples exist for any model with a high input dimension (in relation to the available training data), differentiability only helps with finding them.

Re: Robust Adversarial Examples

#34

Post by larsiusprime »

IanCal wrote: post #20
hyperion2010 wrote: post #18

My own view of this having spent some time in visual neuroscience is that if you really want vision that is robust to these kinds of issues then you have to build a geometric representation of the world first, and then learn/map categories from that. Trying to jump from a matrix to a label without having an intervening topological/geometric model of the world in between (having 2 eyes and/or the ability to move and h…

Some disjointed thoughts. > we think we are recapitulating biology when in fact we are doing nothing of the sort (as these adversarial examples reveal beautifully). I'm not sure I'd go so far. There's a pretty long list of optical illusions. Seeing motion where there clearly is none, not comparing distances correctly and most relevant here is things that look like a face. Here are a few selected famous examples: http…

I think we should refrain from saying we are recapitulating biology until we have reached the point where the machine systems tend to succeed AND fail in the SAME ways that the biological systems do.

Re: Robust Adversarial Examples

#35

Post by csomar »

arnioxux wrote: post #16

There are plenty of adversarial examples for humans too: http://i.imgur.com/mOTHgnf.jpg

Is it though? The human correctly interpreted the image. The problem is that the image was well, not "real". Human have a limit of figuring out what is real and not real based on experience.

Re: Robust Adversarial Examples

#36

Post by larsiusprime »

arnioxux wrote: post #16

There are plenty of adversarial examples for humans too: http://i.imgur.com/mOTHgnf.jpg

I think the point is that these models are often hyped as being proof that we've reproduced human visual systems, and adversarial examples that humans can still resolve are evidence against that.

When the adversarial examples for humans MATCH the adversarial examples for image classifiers, that would be evidence of having reproduced a biological system.

Re: Robust Adversarial Examples

#39

Post by [deleted] »

yorwba wrote: post #31
DonbunEf7 wrote: post #15

Earlier quoted context omitted.

You are, unfortunately, probably just playing out Mr. Crab's obsession with record players. Remember that these tricky images are based on the principle that machine-learning algorithms are differentiable and high-dimensional . There is a lot of ways to transition between, say, the desktop dimension and the cat dimension, and it's all continuous, so we're guaranteed to be able to influence the machine in that sort of…

But we don't really care about the cat-masquerading-as-desktop category in itself, so an adversarial example that makes a cat look like a cat-masquerading-as-desktop, or masquerades a cat-masquerading-as-desktop as a cat, isn't really relevant. By adding enough adversarial examples to the training set, you can absolutely immunize a model against adversarial perturbations of the training data. The problem is that the…

[deleted]
Post reply on HN