Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

Post reply on HN
Print view View original

211–220 of 220 posts

Re: Images that fool computer vision raise security concerns

#211

Post by jeffclune »

oh_sigh wrote: post #202
karpathy wrote: post #122

This work has led to some unfortunate misconceptions. In particular, this weakness has nothing to do with Computer Vision and also nothing to do with deep learning. They only break ConvNets on images because images are fun to look at and ConvNets are state of the art. But at its core, the weakness is related to use of linear functions. In fact, you can break a simple linear classifier (e.g. Softmax Classifier or Logi…

Of course, this requires that the attacker have complete knowledge of all the weights of the given network.

No, it does not. See my reply to similar comments.

Re: Images that fool computer vision raise security concerns

#212

Post by joe_the_user »

JoeAltmaier wrote: post #209
joe_the_user wrote: post #194

Earlier quoted context omitted.

Can they create human equivalents? That would require that human vision works in the fashion of neural networks/svms/etc. There actually isn't any evidence for this.

What?! Neural networks are explicitly designed to work in the fashion of - you guessed it, neural networks. Like, retina and optic nerve.

"In modern software implementations of artificial neural networks, the approach inspired by biology has been largely abandoned for a more practical approach based on statistics and signal processing." [1]

[1] http://en.wikipedia.org/wiki/Artificial_neural_network

Re: Images that fool computer vision raise security concerns

#213

Post by kingkawn »

rndn wrote: post #207
kingkawn wrote: post #61

Maybe this is the algorithm making art

I quite like this thought, though art is about combining known patterns in a novel way. Here they create unknown patters to evoke associations with known but unrelated patterns. It's kind of reverse-art.

I think art is about the relationship of the viewer to the work, not about the explanation for the work's creation. The viewer is the place where it all goes down.

Re: Images that fool computer vision raise security concerns

#214

Post by JoeAltmaier »

joe_the_user wrote: post #212
JoeAltmaier wrote: post #209

Earlier quoted context omitted.

What?! Neural networks are explicitly designed to work in the fashion of - you guessed it, neural networks. Like, retina and optic nerve.

"In modern software implementations of artificial neural networks, the approach inspired by biology has been largely abandoned for a more practical approach based on statistics and signal processing." [1] [1] http://en.wikipedia.org/wiki/Artificial_neural_network

The embodiment is changed of course. But the process of successive ranks of weighted accumulators has not been. Which is the neural model. Of course its different. But there's still the question of, could failure modes of the mathematical model be present in the biological one? Its a question of modeling, not wetware vs hardware.

Re: Images that fool computer vision raise security concerns

#215

Post by JoeAltmaier »

msandford wrote: post #3

This is some excellent research! It reminds me of the CV dazzle anti-facial-recognition makeup that made the rounds a while ago: http://www.theatlantic.com/features/archive/2014/07/makeup/3... This definitely reinforces my belief that having humans in the loop is not only desirable but necessary. For the majority of human history minus a few years you could only be accused of a crime by another human being. I'd like…

Humanity isn't some static thing. Cultures become dysfunctional just like familes. Sometimes they have to be changed or they will collapse. Its disingenuous to suggest 'everybody would be happy if only the cops would quit hassling me'. That's sophomoric.

Re: Images that fool computer vision raise security concerns

#216

Post by joe_the_user »

JoeAltmaier wrote: post #214
joe_the_user wrote: post #212

Earlier quoted context omitted.

"In modern software implementations of artificial neural networks, the approach inspired by biology has been largely abandoned for a more practical approach based on statistics and signal processing." [1] [1] http://en.wikipedia.org/wiki/Artificial_neural_network

The embodiment is changed of course. But the process of successive ranks of weighted accumulators has not been. Which is the neural model. Of course its different. But there's still the question of, could failure modes of the mathematical model be present in the biological one? Its a question of modeling, not wetware vs hardware.

But that model is not the way the overall activity of the brain is currently understood - neurons are seen as being far more complex than simple threshold machines. They may involve thresh effects but the claim of them working overall like any version of artificial neural works is no longer supported by anyone.

Re: Images that fool computer vision raise security concerns

#217

Post by tripzilch »

yosinski wrote: post #171
JoeAltmaier wrote: post #154

Earlier quoted context omitted.

Perhaps a single experiment on millions of different people? A web experiment of some kind? "Which image looks more like a panda?" and flash two images on the screen.

That's a good idea, though note that there's a difference between asking "Which of these two images looks more like a panda?" and "Which of these two images looks more like a panda than a dog or cat?". The latter is the supervised learning setting used in the paper, and generally could lead to examples that look very different than pandas, as long as they look slightly more like pandas than dogs or cats. The former m…

Problem is, even if you succeed and end up with a fabricated picture that fools human neural nets into believing it's a picture of a panda, how would you tell it's not really a picture of a panda?

You'd need another classifier to tell you "nope it's actually just random noise and shapes" ... hm.

Re: Images that fool computer vision raise security concerns

#218

Post by JoeAltmaier »

tripzilch wrote: post #217
yosinski wrote: post #171

Earlier quoted context omitted.

That's a good idea, though note that there's a difference between asking "Which of these two images looks more like a panda?" and "Which of these two images looks more like a panda than a dog or cat?". The latter is the supervised learning setting used in the paper, and generally could lead to examples that look very different than pandas, as long as they look slightly more like pandas than dogs or cats. The former m…

Problem is, even if you succeed and end up with a fabricated picture that fools human neural nets into believing it's a picture of a panda, how would you tell it's not really a picture of a panda? You'd need another classifier to tell you "nope it's actually just random noise and shapes" ... hm.

Probably not hard - just engage the higher cognitive functions. "Does it look like noise? Yeah." Or just close one eye and look again.

Re: Images that fool computer vision raise security concerns

#219

Post by tripzilch »

JoeAltmaier wrote: post #218
tripzilch wrote: post #217

Earlier quoted context omitted.

Problem is, even if you succeed and end up with a fabricated picture that fools human neural nets into believing it's a picture of a panda, how would you tell it's not really a picture of a panda? You'd need another classifier to tell you "nope it's actually just random noise and shapes" ... hm.

Probably not hard - just engage the higher cognitive functions. "Does it look like noise? Yeah." Or just close one eye and look again.

I think you missed my somewhat deeper philosophical point :)

Who gets to decide what is really a picture of a panda?

If we'd manage to craft a picture that could with very high certainty trick human neural nets (for the sake of argument, including those higher cognitive functions) into believing something is a picture of a panda, "except it actually really isn't", what does that even mean?

Human insists it's a picture of a panda, computer classifier maintains it's noise and shapes.

Who is right? :)

Re: Images that fool computer vision raise security concerns

#220

Post by JoeAltmaier »

tripzilch wrote: post #219
JoeAltmaier wrote: post #218

Earlier quoted context omitted.

Probably not hard - just engage the higher cognitive functions. "Does it look like noise? Yeah." Or just close one eye and look again.

I think you missed my somewhat deeper philosophical point :) Who gets to decide what is really a picture of a panda? If we'd manage to craft a picture that could with very high certainty trick human neural nets (for the sake of argument, including those higher cognitive functions) into believing something is a picture of a panda, "except it actually really isn't" , what does that even mean? Human insists it's a pictu…

Interesting, sure. But I started out wondering if some obviously-noise picture could be found that fooled humans, at least at first glance. "Hey a panda! Wait, what was I thinking, that's just noise!" It would be weird and cool, on the order of the dress meme etc. but much more so.

Kind of like the memes in Snowcrash, ancient forgotten symbols that make up the kernel of human thought.

Post reply on HN