Robust Adversarial Examples • Hacker News Board

Re: Robust Adversarial Examples

#41

Post by mannykannot » Tue, Jul 18, 2017, 2:24 PM UTC

yorwba wrote: post #31

DonbunEf7 wrote: post #15

Earlier quoted context omitted.

You are, unfortunately, probably just playing out Mr. Crab's obsession with record players. Remember that these tricky images are based on the principle that machine-learning algorithms are differentiable and high-dimensional . There is a lot of ways to transition between, say, the desktop dimension and the cat dimension, and it's all continuous, so we're guaranteed to be able to influence the machine in that sort of…

But we don't really care about the cat-masquerading-as-desktop category in itself, so an adversarial example that makes a cat look like a cat-masquerading-as-desktop, or masquerades a cat-masquerading-as-desktop as a cat, isn't really relevant. By adding enough adversarial examples to the training set, you can absolutely immunize a model against adversarial perturbations of the training data. The problem is that the…

We care about the existence of a nontrivial set of images that demonstrate a troubling lack of robustness in image classifiers, at least until we we have good reason to say with confidence that such failures will not be a problem in practice.

Re: Robust Adversarial Examples

#42

Post by andbberger » Tue, Jul 18, 2017, 3:19 PM UTC

yakult wrote: post #25

azag0 wrote: post #23

Earlier quoted context omitted.

This is a huge topic in applying ML in physics and chemistry where we already have a lot of prior detailed knwoledge about the systems we want to describe and it would be silly not to build it into the ML models.

What's the current state of art in this direction? Is there a way to encode equations explicitly prior to training?

Variational inference

Re: Robust Adversarial Examples

#43

Post by landon32 » Tue, Jul 18, 2017, 3:51 PM UTC

clickok wrote: post #21

hyperion2010 wrote: post #18

My own view of this having spent some time in visual neuroscience is that if you really want vision that is robust to these kinds of issues then you have to build a geometric representation of the world first, and then learn/map categories from that. Trying to jump from a matrix to a label without having an intervening topological/geometric model of the world in between (having 2 eyes and/or the ability to move and h…

We tried that; the reason deep nets are popular is that they outperform geometric (or other problem-specific) models. This might be because they implicitly develop such representations somewhere along the way, or because such a representation is not really necessary for visual classification. Additionally, introducing ancillary modules is not without cost-- you might gain robustness to some kinds of adversarial input…

have we tried using DL to construct 3d space, using either video or 3d images? That seems closer to what humans do than just the 2d images

Re: Robust Adversarial Examples

#44

Post by landon32 » Tue, Jul 18, 2017, 3:58 PM UTC

yorwba wrote: post #33

mamp wrote: post #22

Earlier quoted context omitted.

You assume that all sensors are working on 'images' and that the algorithm are all using gradient descent.

Replace 'images' with 'sensor data' and adversarial examples can still be generated. They might not be as easy to feed into the vehicles hardware (e.g. requiring speakers to fool an acoustic sensor), but the same principles apply. It's also not necessary for the recognition algorithms to be using gradient descent, so long as they are differentiable (or can be approximated by a model that is), you can use gradient des…

That's really interesting though, you might be able to combine something like Random Forest with Neural Nets to make them more robust to adversarial images.

Re: Robust Adversarial Examples

#45

Post by mring33621 » Tue, Jul 18, 2017, 4:01 PM UTC

mannykannot wrote: post #40

sharemywin wrote: post #37

To me it's an image of a picture regardless of the contents of the picture.

A classifier that implements that logic is not going to be useful for anything.

NOT PICTURE

Re: Robust Adversarial Examples

#46

Post by sharemywin » Tue, Jul 18, 2017, 4:43 PM UTC

mannykannot wrote: post #40

sharemywin wrote: post #37

To me it's an image of a picture regardless of the contents of the picture.

A classifier that implements that logic is not going to be useful for anything.

It's sure as hell not a cat. cat's aren't flat and don't have white borders. Well I guess it could be road kill.

Once you figure out it's a picture(or photo) of something then figure out what it's a picture of.

Re: Robust Adversarial Examples

#47

Post by pvillano » Tue, Jul 18, 2017, 4:56 PM UTC

I don't know how you guys think this is an adversarial example. I see a picture of a desktop computer.

Re: Robust Adversarial Examples

#48

Post by randyrand » Tue, Jul 18, 2017, 7:42 PM UTC

therajiv wrote: post #2

It's not clear to me how malicious actors can manipulate this observation to confuse self-driving cars. That said, I don't think this discredits the point of the article; it's important to note how easily deep learning models can be fooled if you understand the math behind them. I just think the example of tricking self-driving cars is difficult to relate with / understand.

Why limit yourself to self driving cars? A smart malicious actor would just throw oil out their window on a highway. Watch all the cars crash!!

I think these adversarial examples are near irrelevant issues for self driving cars. If someone does something bad, we prosecute them. Its the same whether you're throwing oil onto a highway, covering up stop signs with adversarial stop signs, or whatever you might want to do.

Now if there was an exploit that caused all self driving cars in the whole country to suddenly crash into walls, that would be one thing. But these image-based attacks are limited to a single intersection or road at time. And after a single car crashes, the intersection gets closed. So if you really want to kill a few people, why not just go and stab them in the neck?

Re: Robust Adversarial Examples

#49

Post by [deleted] » Tue, Jul 18, 2017, 9:06 PM UTC

randyrand wrote: post #28

0xbear wrote: post #4

Earlier quoted context omitted.

You could wear special adversarial clothing for example, or even just project adversarial images onto pavement, walls, poles, road signs, and other reflective surfaces.

you could also throw nails onto a highway out your car window. I'm not sure why someone would though.

[deleted]

Re: Robust Adversarial Examples

#50

Post by std_throwaway » Wed, Jul 19, 2017, 6:45 PM UTC

clickok wrote: post #19

std_throwaway wrote: post #17

Does this effect carry over to classifiers which were trained with different training data?

I am unsure what you mean-- do you mean with different training sets but the same testing set? It's an interesting question; maybe the reason for (some) of these adversarial vulnerabilities is due to a handful of bad training examples. You could formulate it as a search problem to see if there's particular images (or small groups of images) that are responsible for the adversarial vulnerabilities. This might then ind…

> I am unsure what you mean-- do you mean with different training sets but the same testing set?

Yes, assuming we have 10000 different training images. Divide these into 5 sets of 2000 each and train 5 networks with them. Assuming that 2000 images are plenty for this application, we will have 5 well trained networks that have similar performance for a test set.

BUT

They will work slightly differently internally and those "inverse gradient search" methods (or what they are called) might only be able to manipulate an image for one network at the time with "specifically chosen additive noise" while the other 4 are unimpressed.

That's assuming that the manipulation can't be targeted at all 5 classifiers at the same time.