Here's a link to the Yosinski's paper.
http://yosinski.com/media/papers/Nguyen__2014__arXiv__Deep_N...
Images that fool computer vision raise security concerns
21–30 of 220 posts
Re: Images that fool computer vision raise security concerns
#22If you look really closely at the noisy images, you can see little blotches that vaguely resemble the things the computer recognized them as.
Re: Images that fool computer vision raise security concerns
#23The pattern-based illusions are actually quite interesting, almost artistic. Half of them are recognizable to humans, the other half at least make sense when identified. I wonder if we can automate the production of postmodern art :)
Deep Neo Neo Cubism?
Re: Images that fool computer vision raise security concerns
#24Not directly related but, I was at a security related convention and overheard some people talking about an image that when occupying <3/4 of a frame will crash any digital camera. (Phone, DSLR, IP Camera) Does anyone know anymore information about this image and effect? I imagine it's a bug in some low level firmware of a common IC for digital photography DSP but, I'm very unfamiliar with digital cameras. It also could have been complete bunk because I've not heard of it since and it wasn't being showcased at the convention.
Re: Images that fool computer vision raise security concerns
#25The whole idea is to tear down algorithm, simulate unnatural image that triggers the required responses and pass it to classifier. There is no direct link to security risks here.
Re: Images that fool computer vision raise security concerns
#26If you ever work with classifier training, these results are not surprising. You can take all false positives generate by classifier, average them and you will come up with an image that resembles the object to be recognized.
Re: Images that fool computer vision raise security concerns
#27This is some excellent research! It reminds me of the CV dazzle anti-facial-recognition makeup that made the rounds a while ago: http://www.theatlantic.com/features/archive/2014/07/makeup/3... This definitely reinforces my belief that having humans in the loop is not only desirable but necessary. For the majority of human history minus a few years you could only be accused of a crime by another human being. I'd like…
> If everyone is breaking so many laws that the police and courts can't keep up it doesn't mean that humanity is broken. It means that the law has gotten so far out of sync with humanity that the law is broken. People make the laws, not the other way around. The world would be a much better place if more people realized this.
This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial.
If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypassing trials is just a way to cover your ears and shout "la la la" to ignore the problem.
Re: Images that fool computer vision raise security concerns
#28This raises an interesting question. Do we want computers to see "correctly", or to see how we see? Would a preferred computer vision system experience the Checker shadow illusion? http://en.wikipedia.org/wiki/Checker_shadow_illusion If yes, computer vision will be as fallible as ours. If no, then there will always be examples, like the ones presented, where computers will see something different than humans.
A computer vision system can have multiple ways of processing an image. So at the limit, it could interpret a scene in terms of what a human sees and also have a separate, better understanding of the scene.
Re: Images that fool computer vision raise security concerns
#29It is good to know that they need access to a lot of predictions from a net, before they can create an image that will "fool" the net, but look alien to humans. Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets (…
"Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images." Trivially "solved" by treating the ensemble as a single object, then constructing a counterexample. My intuition suggests that while the resulting "fooled you" image may very slowly converge on something human recoginizabl…
Now I have to try this out. My intuition tells me it becomes increasingly hard to create a fooling image, which looks alien, and is able to fool all the nets in the ensemble, even though they have different settings and params. I think they can only fool one net at a time, and have to get very lucky to be able to evolve the image for the other nets, while keeping the same classification. You can't "train" these images on all nets at once, by simply treating the ensemble output as a single net.
If your intuition is right though, then the ensemble may be able to counter with a random selection of nets for its vote: You'd need to evolve images for every possible combination and/or account for nets added in the future.
Re: Images that fool computer vision raise security concerns
#30This raises an interesting question. Do we want computers to see "correctly", or to see how we see? Would a preferred computer vision system experience the Checker shadow illusion? http://en.wikipedia.org/wiki/Checker_shadow_illusion If yes, computer vision will be as fallible as ours. If no, then there will always be examples, like the ones presented, where computers will see something different than humans.
A computer vision system can have multiple ways of processing an image. So at the limit, it could interpret a scene in terms of what a human sees and also have a separate, better understanding of the scene.
That would be interesting; it could flag inputs that are ambiguous to humans but not machines (or vice versa, or when there's a discrepancy at all) since it could suggest that something shady is happening.