Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

Post reply on HN
Print view View original

161–170 of 220 posts

Re: Images that fool computer vision raise security concerns

#161

Post by dunstad »

im2w1l wrote: post #107
lars wrote: post #52

Earlier quoted context omitted.

That isn't quite right. With a sufficiently powerful formal system, you're forced to either have inconsistency or incompleteness - you're describing a system that is inconsistent. It's usually much better to have consistency and to sacrifice completeness. Then you'll have Ps that are true but unprovable, but at least you won't have P=~P which makes the system rather useless.

What does true but unprovable mean? What happens if you take such a proposition, negate it and add as an axiom?

My understanding of the incompleteness theorem is that, for a given set of axioms, there will be unprovably true things. Changing the axioms would change which things were unprovable.

That being said, here is a much better resource than I am: http://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_t...

Re: Images that fool computer vision raise security concerns

#162

Post by [deleted] »

im2w1l wrote: post #107
lars wrote: post #52

Earlier quoted context omitted.

That isn't quite right. With a sufficiently powerful formal system, you're forced to either have inconsistency or incompleteness - you're describing a system that is inconsistent. It's usually much better to have consistency and to sacrifice completeness. Then you'll have Ps that are true but unprovable, but at least you won't have P=~P which makes the system rather useless.

What does true but unprovable mean? What happens if you take such a proposition, negate it and add as an axiom?

[deleted]

Re: Images that fool computer vision raise security concerns

#163

Post by Nvn »

jeffclune wrote: post #146
Houshalter wrote: post #9

A paper came out that explains this effect and a method of minimizing it: http://arxiv.org/abs/1412.6572 Basically neural networks and many other machine learning methods are highly linear and continuous. So changing an input just slightly should change the output just slightly. If you change all of the inputs slightly in just the right directions, you can manipulate the output arbitrarily. These images are highly op…

You say that these images are highly optimized to produce this effect and would not occur by chance, but have you looked at the images in the "fooling" paper? http://www.evolvingai.org/fooling Some of them are very simple, and DO occur a lot in the world. For example, the alternating yellow and black line pattern would be encountered by a driverless car, and it would think it is seeing a school bus.

>Some of them are very simple, and DO occur a lot in the world. For example, the alternating yellow and black line pattern would be encountered by a driverless car, and it would think it is seeing a school bus.

While the image shows a yellow and black line pattern to us, are you sure this is also what the CNN "sees"? Couldn't this image just be the same as the adversarial images, i.e. it responds to many small input values rather than the overall pattern?

If it's possible to make the CNN predict an ostrich for an image of a car, then the same can be done of an image of an alternating yellow and black line pattern, no?

Re: Images that fool computer vision raise security concerns

#164

Post by Lawtonfogle »

warfangle wrote: post #98
Mc_Big_G wrote: post #86

Earlier quoted context omitted.

>>Were there even the slimmest chance of acquittal, few defendants would utter that phrase without there being either a benefit to owning up, or an extra penalty for not doing so Oh man, that's so wrong. Even where there is a LARGE change of acquittal, many choose a plea bargain because they cannot afford a good attorney or because the prosecutor is threatening them with something crazy like 40 years for downloading…

And some choose suicide over the plea bargain.

For some charges, even full aquital isn't enough to recover your life.

Re: Images that fool computer vision raise security concerns

#165

Post by Lawtonfogle »

bmm6o wrote: post #83
frobozz wrote: post #74

Earlier quoted context omitted.

Why should every crime go to trial? If the offender says "Fair cop, guv, you've got me bang to rights", why waste everyone's time proving that (s)he did it? It's basically the same as setting up your conditions to take advantage of short-circuit evaluation. You don't put the time-consuming and resource-hungry part first. Were there even the slimmest chance of acquittal, few defendants would utter that phrase without…

The issue isn't whether every case should go to trial. A plea bargain is an exchange of (forgoing) a trial for a reduced charge or sentence. The undesirable outcome (for pretty much everyone but the prosecutor) is that an innocent person accepts a plea bargain to avoid the worst outcome, and we should do more to prevent this from happening. > If there's a suspicion of false confession, meaning that the suspect may no…

>The issue isn't whether every case should go to trial. A plea bargain is an exchange of (forgoing) a trial for a reduced charge or sentence.

In reality it becomes a punishment for demanding a fair trial.

Re: Images that fool computer vision raise security concerns

#166

Post by jessaustin »

toomuchtodo wrote: post #143
chc wrote: post #129

Earlier quoted context omitted.

Play it out in your head. An elected official declares he wants to try and starve a powerful and relatively popular union. What happens next?

I didn't say it was pleasant. What's necessary rarely is.

So you're advocating some sort of violent overthrow? We can't just snap our fingers and remake the political system into something completely different than it is, no matter how much Lawrence Lessig wishes we could. Grim fanatical purity is good for fundraising, but it won't get any laws passed.

Re: Images that fool computer vision raise security concerns

#167

Post by yosinski »

karpathy wrote: post #151
yosinski wrote: post #136

Earlier quoted context omitted.

> This work has led to some unfortunate misconceptions. Agreed; the weaknesses reported should definitely not be taken to affect only convnets or only deep learning. Ian's "Explaining and Harnessing Adversarial Examples" paper (linked by @Houshalter) should be required reading :). > backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern ou…

The two classes that you describe: 1. Adversary has the weights and architecture and 2. Adversary can only do forward pass and observe output, are equivalent when all you're trying to do is compute the gradient on the data. In case 1 I use backprop, in case 2 I can compute the gradient numerically, it just takes a bit longer. Your stochastic search speeds this up. Likewise, I was not very surprised that you can produ…

> ...in case 2 I can compute the gradient numerically, it just takes a bit longer.

Yep, true, might just take a while. On the other hand, even a very noisy estimate of the gradient might suffice, which could be faster to obtain. Perhaps someone will do that experiment soon. Maybe you could convince one of those students of yours to do this for extra credit?? ;).

> Likewise, I was not very surprised that you can produce fooling images, but it is surprising and concerning that they generalize across models.

Ditto x2.

> It seems that there are entire, huge fooling subspaces of the input space, not just fooling images as points. And that these subspaces overlap a lot from one net to another, likely since they share similar training data (?) unclear.

Yeah. I wonder if the subspaces found using non-gradient based exploration end up being either larger or overlapping more between networks than those found (more easily) with the gradient. Would be another interesting followup experiment.

Re: Images that fool computer vision raise security concerns

#168

Post by yosinski »

jeffclune wrote: post #157
karpathy wrote: post #151

Earlier quoted context omitted.

The two classes that you describe: 1. Adversary has the weights and architecture and 2. Adversary can only do forward pass and observe output, are equivalent when all you're trying to do is compute the gradient on the data. In case 1 I use backprop, in case 2 I can compute the gradient numerically, it just takes a bit longer. Your stochastic search speeds this up. Likewise, I was not very surprised that you can produ…

> Likewise, I was not very surprised that you can produce fooling images, but it is surprising and concerning that they generalize across models. It seems that there are entire, huge fooling subspaces of the input space, not just fooling images as points. And that these subspaces overlap a lot from one net to another, Agreed. That is surprising, and also increases the security risks, because I can produce images on m…

> Agreed. That is surprising, and also increases the security risks, because I can produce images on my in-house network and then take them out into the world to fool other networks without even having access to the outputs of those networks.

Good point. You could also do this with the gradient version too (fool in-house using gradients -> hopefully fool someone else's network), but the transferability of fooling examples might differ depending on how they are found.

Re: Images that fool computer vision raise security concerns

#169

Post by toomuchtodo »

jessaustin wrote: post #166
toomuchtodo wrote: post #143

Earlier quoted context omitted.

I didn't say it was pleasant. What's necessary rarely is.

So you're advocating some sort of violent overthrow? We can't just snap our fingers and remake the political system into something completely different than it is, no matter how much Lawrence Lessig wishes we could. Grim fanatical purity is good for fundraising, but it won't get any laws passed.

No, I'm advocating for politicians to make decisions that are better for society, but will cause them to not be re-elected (similar to what happened in Australia when they outlawed firearms).

Re: Images that fool computer vision raise security concerns

#170

Post by logfromblammo »

chc wrote: post #129
toomuchtodo wrote: post #92

Earlier quoted context omitted.

Can we not just defund them? Can't "Stave The Beast" be used for good?

Play it out in your head. An elected official declares he wants to try and starve a powerful and relatively popular union. What happens next?

Are we talking about Wisconsin governor Scott Walker? Because he survives a recall vote and then goes on to help make Wisconsin a right-to-work state.

But wait. He specifically exempted police unions from his busting legislation, didn't he?

Post reply on HN