Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

Post reply on HN
Print view View original

201–210 of 220 posts

Re: Images that fool computer vision raise security concerns

#201

Post by wodenokoto »

yosinski wrote: post #168
jeffclune wrote: post #157

Earlier quoted context omitted.

> Likewise, I was not very surprised that you can produce fooling images, but it is surprising and concerning that they generalize across models. It seems that there are entire, huge fooling subspaces of the input space, not just fooling images as points. And that these subspaces overlap a lot from one net to another, Agreed. That is surprising, and also increases the security risks, because I can produce images on m…

> Agreed. That is surprising, and also increases the security risks, because I can produce images on my in-house network and then take them out into the world to fool other networks without even having access to the outputs of those networks. Good point. You could also do this with the gradient version too (fool in-house using gradients -> hopefully fool someone else's network), but the transferability of fooling exa…

I've quite enjoyed reading your paper since it was uploaded to arxiv in December and I have been toying with redoing the MNIST part of your experiment on various classifiers. (I'm particularly interested to see if images generated against an SVM can fool a nearest neighbor or something like that.)

But I'm having problems generating images: A top SVM classifier on MNIST has a very stable confidence distribution against noisy images. If I generate 1000 random images, only 1 or 2 of them will have confidences that are different from the median confidence distribution. That is, all the images are classified with the same confidence as class 1. They also share the same confidence for class 2, etc.

So it is very difficult to make changes that affect the output of the classifier.

Any tips on how to get started with generating the images?

Re: Images that fool computer vision raise security concerns

#202

Post by oh_sigh »

karpathy wrote: post #122

This work has led to some unfortunate misconceptions. In particular, this weakness has nothing to do with Computer Vision and also nothing to do with deep learning. They only break ConvNets on images because images are fun to look at and ConvNets are state of the art. But at its core, the weakness is related to use of linear functions. In fact, you can break a simple linear classifier (e.g. Softmax Classifier or Logi…

Of course, this requires that the attacker have complete knowledge of all the weights of the given network.

Re: Images that fool computer vision raise security concerns

#203

Post by EdwardDiego »

a-dub wrote: post #138

Cool! Reminds me of "Shazam Decoys" where barely audible or inaudible energy can be added to a signal to fool Shazam into identifying it as the wrong track. I've often thought there would be an awesome opportunity in there to make a hilarious app that catches cheaters during the music round of Pub Quiz.

> I've often thought there would be an awesome opportunity in there to make a hilarious app that catches cheaters during the music round of Pub Quiz.

The best pub quizzes dim the lights so that smartphone cheats beam forth their sneakiness.

Re: Images that fool computer vision raise security concerns

#204

Post by code_duck »

jessaustin wrote: post #67
happyscrappy wrote: post #54

Earlier quoted context omitted.

> you're making too many criminals 80% of property crime goes unsolved, so unless you legalize vandalism and theft there are going to be criminals.

Perhaps this is another way out of our problem? We could encourage police to "solve" property crimes rather than drug crimes? In analogy to their awful drug-war asset seizures, perhaps they could get a cut of items recovered/reimbursed? I'm sure they would eventually twist such an incentive structure into something else that's awful and unconscionable, but at least in the meantime they might stop with the SWAT raids?

Their solution would probably be to conduct SWAT raids and roadblocks to find stolen property, continuing the trend of constitutionally questionable practices.

Re: Images that fool computer vision raise security concerns

#205

Post by DataCreep »

wwweston wrote: post #193
chongli wrote: post #190

Earlier quoted context omitted.

The inefficiency stems from the complexity of the law. We have built up over many years an unfathomably large codex and an equally staggering infrastructure devoted to training legions of people (lawyers) in how to read, interpret and apply it. How do we address this? One emerging solution employs advanced AIs to pore over the reams of evidence in order to better inform lawyers of the legal situation. I can imagine e…

> The inefficiency stems from the complexity of the law. Not just that, though -- it also comes from the inherent inefficiencies in trying to recover exactly what happened from any given situation in which the question of whether a crime was committed. We could start recording everything that happens, but... that's also a potentially terrifying possibility.

> We could start recording everything that happens

You're in luck! Facebook, Google, and many of the wonderful, selfless people who post on HN are already working on that... or at least defending others' "right" to do so if they aren't doing it themselves.

Various entities are tracking who you know, who you sleep with (http://www.whosdrivingyou.org/blog/ubers-deleted-rides-of-gl...), what your face looks like, what your friends' faces look like (thanks, photo tagging enthusiasts!), who you talk to, what you say to them, when you say it, where you go (thanks to various wonderful sources, including ALPR companies like Vigilant), how long you stay there, what you eat, what you wear, what you watch, what you listen to, where you move your mouse while viewing websites, who your doctor is, what medications you take, what the symptoms of that last rash you had were, what your political views are, what you read, where you work, how many steps you took today, what websites you visit, and about 30,000 other bits of data... just to keep you safe!

The future is so amazing! I don't know what I would do without a customized advertising experience(tm). Such a drastic improvement over life in the past where people were so bored of advertisements that they chose to avoid watching them! What none of us knew at the time was that we really just wanted to see more relevant advertisements more often, while giving up our privacy for the corporations' greater good! I sleep much more soundly after a solid day of being bombarded with advertisements that teach me to be a better consumer!

Re: Images that fool computer vision raise security concerns

#206

Post by irascible »

JoeAltmaier wrote: post #154
yosinski wrote: post #149

Earlier quoted context omitted.

No, but we did make use of (1) a large number of input -> network -> output iterations, along with (2) precisely measured output values to decide which input to try next. It may not be so easy to experiment in the same way on natural organisms (ethically or otherwise). Of course, if you're as clever as Tinbergen, you might be able to come up with patterns that fool organisms even without (1) or (2): https://imgur.com…

Perhaps a single experiment on millions of different people? A web experiment of some kind? "Which image looks more like a panda?" and flash two images on the screen.

Like a picture of a blue and black dress?

Re: Images that fool computer vision raise security concerns

#207

Post by rndn »

kingkawn wrote: post #61

Maybe this is the algorithm making art

I quite like this thought, though art is about combining known patterns in a novel way. Here they create unknown patters to evoke associations with known but unrelated patterns. It's kind of reverse-art.

Re: Images that fool computer vision raise security concerns

#208

Post by JoeAltmaier »

saalweachter wrote: post #191
JoeAltmaier wrote: post #50

Earlier quoted context omitted.

Cool! And it would be so easy. Just show one of those noisy thumbnails, and see if the human can identify it as a 'panda'

That's actually a fun-sounding research project: train humans to classify those images. I'd be very interested to know if humans could learn to classify either sets of images. They might not look like a 'panda' to us, but there is some underlying pattern that a machine can pick out and apply the arbitrary label 'panda' to. Can a human learn that same pattern?

Maybe so. But I was thinking, maybe there're weird patterns that human pattern-matching will erroneously classify as well. Probably unconsciously - if we look again, we'd say 'huh, that's just a mess of pixels'. So you'd have to flash them briefly and ask for an instant answer or something, to catch the pattern matcher before the cognitive check got done?

Re: Images that fool computer vision raise security concerns

#209

Post by JoeAltmaier »

joe_the_user wrote: post #194
JoeAltmaier wrote: post #139

Earlier quoted context omitted.

Wait - they didn't use knowledge of the neural network internal state to calculate these patterns? Does that mean they could create equivalent images for human beings? What would those look like!

Can they create human equivalents? That would require that human vision works in the fashion of neural networks/svms/etc. There actually isn't any evidence for this.

What?! Neural networks are explicitly designed to work in the fashion of - you guessed it, neural networks. Like, retina and optic nerve.

Re: Images that fool computer vision raise security concerns

#210

Post by jeffclune »

wodenokoto wrote: post #201
yosinski wrote: post #168

Earlier quoted context omitted.

> Agreed. That is surprising, and also increases the security risks, because I can produce images on my in-house network and then take them out into the world to fool other networks without even having access to the outputs of those networks. Good point. You could also do this with the gradient version too (fool in-house using gradients -> hopefully fool someone else's network), but the transferability of fooling exa…

I've quite enjoyed reading your paper since it was uploaded to arxiv in December and I have been toying with redoing the MNIST part of your experiment on various classifiers. (I'm particularly interested to see if images generated against an SVM can fool a nearest neighbor or something like that.) But I'm having problems generating images: A top SVM classifier on MNIST has a very stable confidence distribution agains…

I would just unleash evolution. 1 or 2 in the first generation is a toehold, and from there evolution can begin to do its work. You can also try a larger population (e.g. 2000) and let it run for a while.
Post reply on HN