Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

Post reply on HN
Print view View original

151–160 of 220 posts

Re: Images that fool computer vision raise security concerns

#151

Post by karpathy »

yosinski wrote: post #136
karpathy wrote: post #122

This work has led to some unfortunate misconceptions. In particular, this weakness has nothing to do with Computer Vision and also nothing to do with deep learning. They only break ConvNets on images because images are fun to look at and ConvNets are state of the art. But at its core, the weakness is related to use of linear functions. In fact, you can break a simple linear classifier (e.g. Softmax Classifier or Logi…

> This work has led to some unfortunate misconceptions. Agreed; the weaknesses reported should definitely not be taken to affect only convnets or only deep learning. Ian's "Explaining and Harnessing Adversarial Examples" paper (linked by @Houshalter) should be required reading :). > backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern ou…

The two classes that you describe: 1. Adversary has the weights and architecture and 2. Adversary can only do forward pass and observe output, are equivalent when all you're trying to do is compute the gradient on the data. In case 1 I use backprop, in case 2 I can compute the gradient numerically, it just takes a bit longer. Your stochastic search speeds this up.

Likewise, I was not very surprised that you can produce fooling images, but it is surprising and concerning that they generalize across models. It seems that there are entire, huge fooling subspaces of the input space, not just fooling images as points. And that these subspaces overlap a lot from one net to another, likely since they share similar training data (?) unclear. Anyway, really cool work :)

Re: Images that fool computer vision raise security concerns

#152

Post by lars »

im2w1l wrote: post #107
lars wrote: post #52

Earlier quoted context omitted.

That isn't quite right. With a sufficiently powerful formal system, you're forced to either have inconsistency or incompleteness - you're describing a system that is inconsistent. It's usually much better to have consistency and to sacrifice completeness. Then you'll have Ps that are true but unprovable, but at least you won't have P=~P which makes the system rather useless.

What does true but unprovable mean? What happens if you take such a proposition, negate it and add as an axiom?

I'm not a logician by a long shot, so I probably can't explain that correctly. I think Gödel found a way to make a logical proposition refer to itself, and then found a way to assert provability. He could then construct the sentence "this sentence is not provable". He showed that such a sentence must exists within any system of sufficient power. Thus the system must be self-contradictory (inconsistent), or the sentence must be true (and the system must be incomplete). I'm not sure if such a sentence still refers to itself when negated, so I can't answer the last one.

Re: Images that fool computer vision raise security concerns

#153

Post by jeffclune »

humanfromearth wrote: post #51

They can only fool the DNN because they know their weights. In theory, you could do the same for people's brains. As long as you don't publish the specs of your net you should be fine I guess.

The paper does not use the weights of the network to produce (nearly all of) these images (see the responses to Karpathy above).

Re: Images that fool computer vision raise security concerns

#154

Post by JoeAltmaier »

yosinski wrote: post #149
JoeAltmaier wrote: post #139

Earlier quoted context omitted.

Wait - they didn't use knowledge of the neural network internal state to calculate these patterns? Does that mean they could create equivalent images for human beings? What would those look like!

No, but we did make use of (1) a large number of input -> network -> output iterations, along with (2) precisely measured output values to decide which input to try next. It may not be so easy to experiment in the same way on natural organisms (ethically or otherwise). Of course, if you're as clever as Tinbergen, you might be able to come up with patterns that fool organisms even without (1) or (2): https://imgur.com…

Perhaps a single experiment on millions of different people? A web experiment of some kind? "Which image looks more like a panda?" and flash two images on the screen.

Re: Images that fool computer vision raise security concerns

#157

Post by jeffclune »

karpathy wrote: post #151
yosinski wrote: post #136

Earlier quoted context omitted.

> This work has led to some unfortunate misconceptions. Agreed; the weaknesses reported should definitely not be taken to affect only convnets or only deep learning. Ian's "Explaining and Harnessing Adversarial Examples" paper (linked by @Houshalter) should be required reading :). > backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern ou…

The two classes that you describe: 1. Adversary has the weights and architecture and 2. Adversary can only do forward pass and observe output, are equivalent when all you're trying to do is compute the gradient on the data. In case 1 I use backprop, in case 2 I can compute the gradient numerically, it just takes a bit longer. Your stochastic search speeds this up. Likewise, I was not very surprised that you can produ…

> Likewise, I was not very surprised that you can produce fooling images, but it is surprising and concerning that they generalize across models. It seems that there are entire, huge fooling subspaces of the input space, not just fooling images as points. And that these subspaces overlap a lot from one net to another,

Agreed. That is surprising, and also increases the security risks, because I can produce images on my in-house network and then take them out into the world to fool other networks without even having access to the outputs of those networks.

> likely since they share similar training data (?) unclear.

The original Szegedy et al. paper shows that these sort of examples generalize even to networks trained on different subsets of the data (and with different architectures).

> Anyway, really cool work :)

Thanks. :-)

Re: Images that fool computer vision raise security concerns

#158

Post by mikeash »

dragonwriter wrote: post #144
mikeash wrote: post #114

Earlier quoted context omitted.

I've never seen "victimless crime" defined as diffuse versus focused harm, and I would strongly disagree with describing it as such. For example, dumping mercury into a major river causes extremely diffuse harm, but I'd never describe it as a "victimless crime." In the other direction, smoking weed in private harms nobody except the smoker. The harm is so focused it doesn't even touch anyone besides the offender. Yet…

> In the other direction, smoking weed in private harms nobody except the smoker. This is precisely the point that is contention. The entire argument for prohibiting marijuana is that this is, in fact, not the case, and that, through a number of indirect channels, people "smoking weed in private" harms others throughout society in a variety of ways. Obviously, as I said, there is considerable disagreement about wheth…

I still don't see any connection.

Lots of what I would call "victimless crimes" are outlawed because of what the proponents of criminalization see as focused harm. Prostitution, for example, is seen as either harming the prostitute, or harming the patron's family. For drugs, it's often considered that the harm is focused on the user, not diffuse.

And again, lots of crimes with diffuse harm are generally agreed on to not fall into the "victimless" category, like dumping toxic materials.

Re: Images that fool computer vision raise security concerns

#159

Post by mikeash »

hueving wrote: post #147
mikeash wrote: post #114

Earlier quoted context omitted.

I've never seen "victimless crime" defined as diffuse versus focused harm, and I would strongly disagree with describing it as such. For example, dumping mercury into a major river causes extremely diffuse harm, but I'd never describe it as a "victimless crime." In the other direction, smoking weed in private harms nobody except the smoker. The harm is so focused it doesn't even touch anyone besides the offender. Yet…

Maybe second-order effects is a better way to describe it than 'diffuse'. Take the following: >In the other direction, smoking weed in private harms nobody except the smoker. The harm is so focused it doesn't even touch anyone besides the offender. Yet this is almost the canonical example of "victimless crime." It's true that this doesn't directly harm anyone. However, if the smoker is doing this 'illegally' (without…

As far as I understand this comment, you're saying that smoking weed causes harm because it's illegal, and it's illegal because it causes harm. That's a bit... odd.

Re: Images that fool computer vision raise security concerns

#160

Post by ars »

humanfromearth wrote: post #51

They can only fool the DNN because they know their weights. In theory, you could do the same for people's brains. As long as you don't publish the specs of your net you should be fine I guess.

> In theory, you could do the same for people's brains.

No because the instant you used an image that was close, but wrong the human brain would retrain.

I wonder in the neural net can do the same - try this experiment on an active neural net and let it train itself. (i.e. don't tell it it's being faked, let it figure it out then correct for it).

Post reply on HN