Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

Post reply on HN
Print view View original

121–130 of 220 posts

Re: Images that fool computer vision raise security concerns

#121

Post by platz »

clort wrote: post #112
platz wrote: post #94

Earlier quoted context omitted.

it might not be interesting to you as a researcher, but to me as a participant in in a world increasingly filled with face recognition products it's worth knowing that different products will have different capabilities based on their "speed" requirements.

I am not a researcher, but I would suggest that perhaps you would find that in the case where a crowd was being parsed in realtime (looking out for somebody so we can catch them before they leave the station for example), the "speed" dial would be turned up as high as necessary, but that the video stream would be stored and the data set reprocessed at a later time with it at a better setting (so you can still say "ye…

yes, highlighting what is possible is always needs to be taken into account. But again, what is possible is not the same as what happens in reality. Just because we can't rule something out, does not make it certain either.

> the "speed" dial would be turned up as high as necessary

It's possible, and probable given the situation (i.e. government, terror acts), but are there situations where this wouldn't happen in a different circumstance? Would mall security, or a marketing company operating an advertising product, have the ability, authorization, know-how, or financial incentive to do so if this wasn't a life/death situation?

Not every application of facial recognition technology will be targeted to terror suspects/bombings.

Re: Images that fool computer vision raise security concerns

#122

Post by karpathy »

This work has led to some unfortunate misconceptions.

In particular, this weakness has nothing to do with Computer Vision and also nothing to do with deep learning. They only break ConvNets on images because images are fun to look at and ConvNets are state of the art. But at its core, the weakness is related to use of linear functions. In fact, you can break a simple linear classifier (e.g. Softmax Classifier or Logistic Regression) in just the same way. And you could similarly break speech recognition systems, etc. I covered this in CS231n in "Visualizing/Understanding ConvNets" lecture, slides around #50 (http://vision.stanford.edu/teaching/cs231n/slides/lecture8.p...).

The way I like to think about this is that for any input (e.g. an image), imagine there are billion tiny noise patterns you could add to the input. The vast majority in hundreds of billions are harmless and don't change the classifications, but given the weights of the network, backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern out of all billions.

All that being said, this is a concern and people are working on fixing it.

Re: Images that fool computer vision raise security concerns

#123

Post by crpatino »

Houshalter wrote: post #9

A paper came out that explains this effect and a method of minimizing it: http://arxiv.org/abs/1412.6572 Basically neural networks and many other machine learning methods are highly linear and continuous. So changing an input just slightly should change the output just slightly. If you change all of the inputs slightly in just the right directions, you can manipulate the output arbitrarily. These images are highly op…

Nobody is claiming that this changes are more than astronomically possible to be produced by random processes, the title in such case should have read "... raises robustness concerns" instead of security concerns.

In every security system, it is assumed that if there is an attack surface, sooner or later an intelligent adversary will come and exploit it. And there is a long precedent saying that if you see the word "linear" anywhere in the attack surface description, the adversary is bond to come sooner rather than later.

Re: Images that fool computer vision raise security concerns

#124

Post by eli_gottlieb »

JoeAltmaier wrote: post #37
maxerickson wrote: post #28

Earlier quoted context omitted.

A computer vision system can have multiple ways of processing an image. So at the limit, it could interpret a scene in terms of what a human sees and also have a separate, better understanding of the scene.

The OP shows that computers DO NOT have a better understanding. Its evident they have no understanding at all; they are simply doing math on pixels and latching on to coincidental patters of color or shading. People recognize things by building a 3D model in their head, then comparing that to billions of experiential models, finding a match and then using cognition to test that match. "Is that a bird? No, its just a…

>People recognize things by building a 3D model in their head, then comparing that to billions of experiential models, finding a match and then using cognition to test that match. "Is that a bird? No, its just a pattern of dog dropping smeared on a bench. Ha ha!"

So you're saying people are generative reasoners with very general hypothesis classes rather than discriminative learners with tiny hypothesis classes.

To which the obvious response is, yes, we know that. The question is how to make some computerization of general, generative learning work fast and well.

Re: Images that fool computer vision raise security concerns

#125

Post by sp332 »

JoeAltmaier wrote: post #102
sp332 wrote: post #91

Earlier quoted context omitted.

I don't think your description of how humans recognize things is true. You can do object recognition of a silhouette, or a 2D image, or an impressionistic rendering. http://www.sciencedaily.com/releases/2009/04/090429132231.ht...

We can't help but build real models of what we see - our retina/optic nerve are already doing this before our brain even receives the 'image'! I can't help but believe some of the image recognition mentioned in your article, especially of icons, is built through previous experience with similar iconic images. Symbols for things become associated with the real things. Its a modern adaptation of a much older processing…

OK... but how is that pattern-matching different from what the computer is doing? Why is human pattern-matching "understanding" and computer patter-matching is not?

Re: Images that fool computer vision raise security concerns

#126

Post by appleflaxen »

This makes me wonder if we might be flirting with the computational equivalent of autism.

Could autistic children have learning impairment due to their inability to correctly sort/segregate stimuli, in the same way that these neural networks generate high-confidence false-positives?

Re: Images that fool computer vision raise security concerns

#127

Post by JoeAltmaier »

eli_gottlieb wrote: post #124
JoeAltmaier wrote: post #37

Earlier quoted context omitted.

The OP shows that computers DO NOT have a better understanding. Its evident they have no understanding at all; they are simply doing math on pixels and latching on to coincidental patters of color or shading. People recognize things by building a 3D model in their head, then comparing that to billions of experiential models, finding a match and then using cognition to test that match. "Is that a bird? No, its just a…

>People recognize things by building a 3D model in their head, then comparing that to billions of experiential models, finding a match and then using cognition to test that match. "Is that a bird? No, its just a pattern of dog dropping smeared on a bench. Ha ha!" So you're saying people are generative reasoners with very general hypothesis classes rather than discriminative learners with tiny hypothesis classes. To w…

People are far more than that. Lots of our brain is dedicated to visual modeling. Those 'hypothesis classes' are just the tip of the iceberg. For computers, they're the whole enchilada. To mix metaphors.

Re: Images that fool computer vision raise security concerns

#128

Post by [deleted] »

uncoder0 wrote: post #24

Not directly related but, I was at a security related convention and overheard some people talking about an image that when occupying <3/4 of a frame will crash any digital camera. (Phone, DSLR, IP Camera) Does anyone know anymore information about this image and effect? I imagine it's a bug in some low level firmware of a common IC for digital photography DSP but, I'm very unfamiliar with digital cameras. It also co…

[deleted]

Re: Images that fool computer vision raise security concerns

#129

Post by chc »

toomuchtodo wrote: post #92
jessaustin wrote: post #89

Earlier quoted context omitted.

It's difficult (not impossible!) to directly oppose politically well-connected factions on their bread-and-butter issues. We might have more luck saying to police departments "here is another way you can have lots of money to spend" rather than "we're taking away federal support for the Drug War, which for some time has been the only way you can hire more cops and procure more equipment". The trick would be to make s…

Can we not just defund them? Can't "Stave The Beast" be used for good?

Play it out in your head. An elected official declares he wants to try and starve a powerful and relatively popular union. What happens next?

Re: Images that fool computer vision raise security concerns

#130

Post by geographomics »

uncoder0 wrote: post #24

Not directly related but, I was at a security related convention and overheard some people talking about an image that when occupying <3/4 of a frame will crash any digital camera. (Phone, DSLR, IP Camera) Does anyone know anymore information about this image and effect? I imagine it's a bug in some low level firmware of a common IC for digital photography DSP but, I'm very unfamiliar with digital cameras. It also co…

Not what you're looking for, but your comment reminded me of this: https://en.wikipedia.org/wiki/BLIT_%28short_story%29
Post reply on HN