Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

Post reply on HN
Print view View original

141–150 of 220 posts

Re: Images that fool computer vision raise security concerns

#141

Post by jeffclune »

JoeAltmaier wrote: post #139
yosinski wrote: post #136

Earlier quoted context omitted.

> This work has led to some unfortunate misconceptions. Agreed; the weaknesses reported should definitely not be taken to affect only convnets or only deep learning. Ian's "Explaining and Harnessing Adversarial Examples" paper (linked by @Houshalter) should be required reading :). > backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern ou…

Wait - they didn't use knowledge of the neural network internal state to calculate these patterns? Does that mean they could create equivalent images for human beings? What would those look like!

They would look like optical illusions!

Re: Images that fool computer vision raise security concerns

#142

Post by jeffclune »

yosinski wrote: post #136
karpathy wrote: post #122

This work has led to some unfortunate misconceptions. In particular, this weakness has nothing to do with Computer Vision and also nothing to do with deep learning. They only break ConvNets on images because images are fun to look at and ConvNets are state of the art. But at its core, the weakness is related to use of linear functions. In fact, you can break a simple linear classifier (e.g. Softmax Classifier or Logi…

> This work has led to some unfortunate misconceptions. Agreed; the weaknesses reported should definitely not be taken to affect only convnets or only deep learning. Ian's "Explaining and Harnessing Adversarial Examples" paper (linked by @Houshalter) should be required reading :). > backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern ou…

Ha. Jason and I were typing at the same time. :-)

Re: Images that fool computer vision raise security concerns

#143

Post by toomuchtodo »

chc wrote: post #129
toomuchtodo wrote: post #92

Earlier quoted context omitted.

Can we not just defund them? Can't "Stave The Beast" be used for good?

Play it out in your head. An elected official declares he wants to try and starve a powerful and relatively popular union. What happens next?

I didn't say it was pleasant. What's necessary rarely is.

Re: Images that fool computer vision raise security concerns

#144

Post by dragonwriter »

mikeash wrote: post #114
dragonwriter wrote: post #100

Earlier quoted context omitted.

So-called "victimless crimes" are crimes where the harm perceived is a diffuse harm rather than a focused harm on a particular individual. There's really no reason why diffuse harms should not be criminal while focused ones should (arguably, diffuse harms are less able to be addressed by individual action through the civil justice system, so outside of focused harms where the victim is unable to pursue action after t…

I've never seen "victimless crime" defined as diffuse versus focused harm, and I would strongly disagree with describing it as such. For example, dumping mercury into a major river causes extremely diffuse harm, but I'd never describe it as a "victimless crime." In the other direction, smoking weed in private harms nobody except the smoker. The harm is so focused it doesn't even touch anyone besides the offender. Yet…

> In the other direction, smoking weed in private harms nobody except the smoker.

This is precisely the point that is contention. The entire argument for prohibiting marijuana is that this is, in fact, not the case, and that, through a number of indirect channels, people "smoking weed in private" harms others throughout society in a variety of ways.

Obviously, as I said, there is considerable disagreement about whether this is, in fact, the case, (and, also, as to whether, even if it is, prohibition mitigates or exacerbates these harms, and as to whether, in any case, prohibition is an ethically-acceptable response even if the harms exist and prohibition mitigates them.)

But it certainly is not the case that those who support prohibition generally agree that the crimes are "victimless" that those opposed to prohibition describe that way.

Re: Images that fool computer vision raise security concerns

#145

Post by jeffclune »

compbio wrote: post #10

It is good to know that they need access to a lot of predictions from a net, before they can create an image that will "fool" the net, but look alien to humans. Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets (…

Surprisingly, ensembles do not really help. We have tried this and it does not work (the final paper for CVPR 2015 will show these results). Also see the work of Szegedy et al. and Goodfellow et all, which also show that ensembles do not really help.

Re: Images that fool computer vision raise security concerns

#146

Post by jeffclune »

Houshalter wrote: post #9

A paper came out that explains this effect and a method of minimizing it: http://arxiv.org/abs/1412.6572 Basically neural networks and many other machine learning methods are highly linear and continuous. So changing an input just slightly should change the output just slightly. If you change all of the inputs slightly in just the right directions, you can manipulate the output arbitrarily. These images are highly op…

You say that these images are highly optimized to produce this effect and would not occur by chance, but have you looked at the images in the "fooling" paper?

http://www.evolvingai.org/fooling

Some of them are very simple, and DO occur a lot in the world. For example, the alternating yellow and black line pattern would be encountered by a driverless car, and it would think it is seeing a school bus.

Re: Images that fool computer vision raise security concerns

#147

Post by hueving »

mikeash wrote: post #114
dragonwriter wrote: post #100

Earlier quoted context omitted.

So-called "victimless crimes" are crimes where the harm perceived is a diffuse harm rather than a focused harm on a particular individual. There's really no reason why diffuse harms should not be criminal while focused ones should (arguably, diffuse harms are less able to be addressed by individual action through the civil justice system, so outside of focused harms where the victim is unable to pursue action after t…

I've never seen "victimless crime" defined as diffuse versus focused harm, and I would strongly disagree with describing it as such. For example, dumping mercury into a major river causes extremely diffuse harm, but I'd never describe it as a "victimless crime." In the other direction, smoking weed in private harms nobody except the smoker. The harm is so focused it doesn't even touch anyone besides the offender. Yet…

Maybe second-order effects is a better way to describe it than 'diffuse'. Take the following:

>In the other direction, smoking weed in private harms nobody except the smoker. The harm is so focused it doesn't even touch anyone besides the offender. Yet this is almost the canonical example of "victimless crime."

It's true that this doesn't directly harm anyone. However, if the smoker is doing this 'illegally' (without a medical license, not in WA or CO, etc) and didn't grow it, he/she is participating in and supporting an illegal drug market via increased demand. If the smoker weren't participating, it would reduce the demand that drives smugglers and the horrific things in Mexico.

Re: Images that fool computer vision raise security concerns

#148

Post by jeffclune »

compbio wrote: post #10

It is good to know that they need access to a lot of predictions from a net, before they can create an image that will "fool" the net, but look alien to humans. Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets (…

Also, you don't necessarily need a lot of predictions from a net to fool it, because (another surprising result) images that fool one net tend to fool others! So I can create fooling images on my in-house net and then take them and fool your net-used-for-some-important-application without getting any feedback from your net. That's very surprising, and does raise serious security concerns.

Re: Images that fool computer vision raise security concerns

#149

Post by yosinski »

JoeAltmaier wrote: post #139
yosinski wrote: post #136

Earlier quoted context omitted.

> This work has led to some unfortunate misconceptions. Agreed; the weaknesses reported should definitely not be taken to affect only convnets or only deep learning. Ian's "Explaining and Harnessing Adversarial Examples" paper (linked by @Houshalter) should be required reading :). > backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern ou…

Wait - they didn't use knowledge of the neural network internal state to calculate these patterns? Does that mean they could create equivalent images for human beings? What would those look like!

No, but we did make use of (1) a large number of input -> network -> output iterations, along with (2) precisely measured output values to decide which input to try next. It may not be so easy to experiment in the same way on natural organisms (ethically or otherwise).

Of course, if you're as clever as Tinbergen, you might be able to come up with patterns that fool organisms even without (1) or (2):

https://imgur.com/a/ibMUn

Re: Images that fool computer vision raise security concerns

#150

Post by jeffclune »

adrusi wrote: post #12

The pattern-based illusions are actually quite interesting, almost artistic. Half of them are recognizable to humans, the other half at least make sense when identified. I wonder if we can automate the production of postmodern art :)

In fact, we submitted them to an art competition and they were accepted and put up in a museum!

http://www.evolvingai.org/share/20150213_184537.jpg

Post reply on HN