Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

Post reply on HN
Print view View original

31–40 of 220 posts

Re: Images that fool computer vision raise security concerns

#31

Post by mlmonkey »

Houshalter wrote: post #9

A paper came out that explains this effect and a method of minimizing it: http://arxiv.org/abs/1412.6572 Basically neural networks and many other machine learning methods are highly linear and continuous. So changing an input just slightly should change the output just slightly. If you change all of the inputs slightly in just the right directions, you can manipulate the output arbitrarily. These images are highly op…

> Basically neural networks and many other machine learning methods are highly linear

No they're not! You introduce non-linearities like the sigmoid or tanh to make them highly non-linear.

Re: Images that fool computer vision raise security concerns

#32

Post by randcraw »

This is not so different from recognizing images in their fourier frequency domain. The frequency features and their origins in the spatial domain can be made very unintuitive.

But I'm not clear how important this phenomenon really is to the practice of CV, since 1) 'spoofed' images are highly specific to each DNN being used, and 2) a trivial reality check of the image can always 'out' examples like these.

Re: Images that fool computer vision raise security concerns

#34

Post by Balgair »

z5h wrote: post #7

This raises an interesting question. Do we want computers to see "correctly", or to see how we see? Would a preferred computer vision system experience the Checker shadow illusion? http://en.wikipedia.org/wiki/Checker_shadow_illusion If yes, computer vision will be as fallible as ours. If no, then there will always be examples, like the ones presented, where computers will see something different than humans.

You can always fool any system into a paradox. Very grossly speaking, this comes out of Godel's incompleteness theorem; that with any set of laws you can always get P=~P out of the set. How this paradox looks, acts, or feels is interesting and possibly artistic, as the OP shows. If anything, I think there is a bit of beauty, art, and cleverness in that.

Re: Images that fool computer vision raise security concerns

#35

Post by acadien »

randcraw wrote: post #32

This is not so different from recognizing images in their fourier frequency domain. The frequency features and their origins in the spatial domain can be made very unintuitive. But I'm not clear how important this phenomenon really is to the practice of CV, since 1) 'spoofed' images are highly specific to each DNN being used, and 2) a trivial reality check of the image can always 'out' examples like these.

Your 2nd point is critical, you can filter these images easily before even running them through the DNN. However researchers are also interested in why it is possible to spoof NN's in general. The typical response of 'overfitting' is being questioned.

Also the question is raised as to whether or not new methods of spoofing are possible that aren't so easily detectable.

Re: Images that fool computer vision raise security concerns

#36

Post by Coincoin »

bwross wrote: post #22

If you look really closely at the noisy images, you can see little blotches that vaguely resemble the things the computer recognized them as.

Yeah, they say the algorithm erroneously sees an Amarillo. But wait, I actually see an Amarillo and I actually see a centipede.

Re: Images that fool computer vision raise security concerns

#37

Post by JoeAltmaier »

maxerickson wrote: post #28
z5h wrote: post #7

This raises an interesting question. Do we want computers to see "correctly", or to see how we see? Would a preferred computer vision system experience the Checker shadow illusion? http://en.wikipedia.org/wiki/Checker_shadow_illusion If yes, computer vision will be as fallible as ours. If no, then there will always be examples, like the ones presented, where computers will see something different than humans.

A computer vision system can have multiple ways of processing an image. So at the limit, it could interpret a scene in terms of what a human sees and also have a separate, better understanding of the scene.

The OP shows that computers DO NOT have a better understanding. Its evident they have no understanding at all; they are simply doing math on pixels and latching on to coincidental patters of color or shading.

People recognize things by building a 3D model in their head, then comparing that to billions of experiential models, finding a match and then using cognition to test that match. "Is that a bird? No, its just a pattern of dog dropping smeared on a bench. Ha ha!"

Re: Images that fool computer vision raise security concerns

#38

Post by woodchuck64 »

"“We realized that the neural nets did not encode knowledge necessary to produce an image of a fire truck, only the knowledge necessary to tell fire trucks apart from other classes,” [Yosinski] explained."

This seems markedly different from biological neural networks. Is the difference one of network structure/algorithm or rather the fact that biological neural networks (in human image processing) actually have time and space to learn a lot about each individual image class?

Re: Images that fool computer vision raise security concerns

#39

Post by maxerickson »

JoeAltmaier wrote: post #37
maxerickson wrote: post #28

Earlier quoted context omitted.

A computer vision system can have multiple ways of processing an image. So at the limit, it could interpret a scene in terms of what a human sees and also have a separate, better understanding of the scene.

The OP shows that computers DO NOT have a better understanding. Its evident they have no understanding at all; they are simply doing math on pixels and latching on to coincidental patters of color or shading. People recognize things by building a 3D model in their head, then comparing that to billions of experiential models, finding a match and then using cognition to test that match. "Is that a bird? No, its just a…

How could I have better put So at the limit, it could?

I meant to talk about what some hypothetical future system could do (which I think was a reasonable context given the comment I replied to), not to characterize current systems.

Re: Images that fool computer vision raise security concerns

#40

Post by [deleted] »

mikeash wrote: post #27
toomuchtodo wrote: post #11

Earlier quoted context omitted.

> If everyone is breaking so many laws that the police and courts can't keep up it doesn't mean that humanity is broken. It means that the law has gotten so far out of sync with humanity that the law is broken. People make the laws, not the other way around. The world would be a much better place if more people realized this.

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…

[deleted]
Post reply on HN