Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Robust Adversarial Examples

blog.openai.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Robust Adversarial Examples

#21

Post by clickok »

hyperion2010 wrote: post #18

My own view of this having spent some time in visual neuroscience is that if you really want vision that is robust to these kinds of issues then you have to build a geometric representation of the world first, and then learn/map categories from that. Trying to jump from a matrix to a label without having an intervening topological/geometric model of the world in between (having 2 eyes and/or the ability to move and h…

We tried that; the reason deep nets are popular is that they outperform geometric (or other problem-specific) models. This might be because they implicitly develop such representations somewhere along the way, or because such a representation is not really necessary for visual classification.

Additionally, introducing ancillary modules is not without cost-- you might gain robustness to some kinds of adversarial inputs at the expense of becoming vulnerable to others. There's plenty of ways to fool biological visual systems: c.f. magic-eye posters, optical illusions, or the various exploits described in Lettvin and Pitts' paper "What the Frog's Eye Tells the Frog's Brain".

Re: Robust Adversarial Examples

#22

Post by mamp »

sagebird wrote: post #10
mamp wrote: post #6

Earlier quoted context omitted.

I doubt self driving cars would rely on a single network on a single image source (I hope not!). Robust systems expect that some of the inferences can be mistaken (noisy). That's why you want to run multiple sensor types into different models, and use some kind of mixture of experts +/- probabilistic fusion.

It doesn't matter how many algorithms or sensors are consulted or combined to form judgment. If an attacker can obtain a self driving vehicle's hardware, and if enough tests can be performed per seconds, the attacker can train images that fool it. Your idea is similar to an appeal to security through obscurity. Might work sometimes, but not generally. (Noise does not help, because you can still discover a gradient to…

You assume that all sensors are working on 'images' and that the algorithm are all using gradient descent.

Re: Robust Adversarial Examples

#23

Post by azag0 »

andbberger wrote: post #14
tachyonbeam wrote: post #9

IMO, what these adversarial examples give us is a way to boost training data. We should augment training datasets with adversarial examples, or use adversarial training methods. The resulting networks would only be more robust as a result. As for self-driving cars, this is a good argument for having multiple sensing modalities in addition to visual, such as radar/lidar/sonar, and multiple cameras, infrared in additio…

But at what point do you have to wonder if we're using the wrong basis? And how do you know that augmenting the data with tiny adversarial perturbations won't just leave the network vulnerable in a different direction? It's pretty obvious how to build translational symmetry into a net that's still expressive and easy to train (convolution). But you have to spoon feed CNNs rotational and other symmetries by augmenting…

This is a huge topic in applying ML in physics and chemistry where we already have a lot of prior detailed knwoledge about the systems we want to describe and it would be silly not to build it into the ML models.

Re: Robust Adversarial Examples

#24

Post by SmallDeadGuy »

nullc wrote: post #11
skishore wrote: post #3

Earlier quoted context omitted.

Why do you say that? The first demo they provide shows that the adversarial image, when printed and then manipulated, still fools the algorithm. That means that the example is robust to various affine transformations but also to the per-pixel noise that is a result of a printing something and then viewing it again through a camera. Suppose you were to place an example like that on a stop sign that fooled a car into t…

Though I generally agree with your point, the tree vs stopsign example may not be the best because it would arguably work equally well on humans.

Only if the adversarial image printed doesn't look like the stop sign, though the example in this article shows that it's entirely possible to make an image that just looks like a distorted/badly-printed kitten to a human but completely different to a computer. A similar image for a stop sign might just look like wear in the paint or weird reflections or something but still look like a stop sign to a human.

Re: Robust Adversarial Examples

#25

Post by yakult »

azag0 wrote: post #23
andbberger wrote: post #14

Earlier quoted context omitted.

But at what point do you have to wonder if we're using the wrong basis? And how do you know that augmenting the data with tiny adversarial perturbations won't just leave the network vulnerable in a different direction? It's pretty obvious how to build translational symmetry into a net that's still expressive and easy to train (convolution). But you have to spoon feed CNNs rotational and other symmetries by augmenting…

This is a huge topic in applying ML in physics and chemistry where we already have a lot of prior detailed knwoledge about the systems we want to describe and it would be silly not to build it into the ML models.

What's the current state of art in this direction? Is there a way to encode equations explicitly prior to training?

Re: Robust Adversarial Examples

#26

Post by bsder »

I can paint a road to a tunnel on a mountain side and fool some amount of people. Meep. Meep.

The problem isn't that there are adversarial inputs. The problem is that the adversarial inputs aren't also adversarial (or detectable) to the human visual system.

Re: Robust Adversarial Examples

#27

Post by randyrand »

SmallDeadGuy wrote: post #24
nullc wrote: post #11

Earlier quoted context omitted.

Though I generally agree with your point, the tree vs stopsign example may not be the best because it would arguably work equally well on humans.

Only if the adversarial image printed doesn't look like the stop sign, though the example in this article shows that it's entirely possible to make an image that just looks like a distorted/badly-printed kitten to a human but completely different to a computer. A similar image for a stop sign might just look like wear in the paint or weird reflections or something but still look like a stop sign to a human.

yes but wont we still notice that self driving cars aren't stopping at the stop sign? and we'd investigate

Re: Robust Adversarial Examples

#28

Post by randyrand »

0xbear wrote: post #4
therajiv wrote: post #2

It's not clear to me how malicious actors can manipulate this observation to confuse self-driving cars. That said, I don't think this discredits the point of the article; it's important to note how easily deep learning models can be fooled if you understand the math behind them. I just think the example of tricking self-driving cars is difficult to relate with / understand.

You could wear special adversarial clothing for example, or even just project adversarial images onto pavement, walls, poles, road signs, and other reflective surfaces.

you could also throw nails onto a highway out your car window. I'm not sure why someone would though.

Re: Robust Adversarial Examples

#29

Post by halomru »

randyrand wrote: post #28
0xbear wrote: post #4

Earlier quoted context omitted.

You could wear special adversarial clothing for example, or even just project adversarial images onto pavement, walls, poles, road signs, and other reflective surfaces.

you could also throw nails onto a highway out your car window. I'm not sure why someone would though.

Or throw rocks from bridges into highway traffic. Teenagers occasionally do that

Re: Robust Adversarial Examples

#30

Post by azag0 »

yakult wrote: post #25
azag0 wrote: post #23

Earlier quoted context omitted.

This is a huge topic in applying ML in physics and chemistry where we already have a lot of prior detailed knwoledge about the systems we want to describe and it would be silly not to build it into the ML models.

What's the current state of art in this direction? Is there a way to encode equations explicitly prior to training?

People now try to use ML anywhere and everywhere so it's wild west a little. Three examples: [1] uses a standard neural net to represent a many-body wave function, with all the machinery of quantum mechanics on top of that, and reinforcement learning to find the true ground state. [2] uses a handcrafted neural net, which by construction already takes advantage of a lot of prior knowledge, to directly predict molecular energies. [3] uses a simple kernel ridge regression coupled with a sophisticated handcrafted scheme to automatically construct a good basis (set of features) for a given input, to predict molecular energies.

In all these cases, the ML itself is not the target problem, but only a tool, and most effort goes into figuring out where exactly to use ML as a part of a larger problem, and how to encode prior knowledge, either via feature construction or neural net handcrafting.

[1] http://sci-hub.io/10.1126/science.aag2302

[2] http://sci-hub.io/10.1038/ncomms13890

[3] https://arxiv.org/pdf/1707.04146.pdf

Post reply on HN