Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

Post reply on HN
Print view View original

11–20 of 220 posts

Re: Images that fool computer vision raise security concerns

#11

Post by toomuchtodo »

msandford wrote: post #3

This is some excellent research! It reminds me of the CV dazzle anti-facial-recognition makeup that made the rounds a while ago: http://www.theatlantic.com/features/archive/2014/07/makeup/3... This definitely reinforces my belief that having humans in the loop is not only desirable but necessary. For the majority of human history minus a few years you could only be accused of a crime by another human being. I'd like…

> If everyone is breaking so many laws that the police and courts can't keep up it doesn't mean that humanity is broken. It means that the law has gotten so far out of sync with humanity that the law is broken. People make the laws, not the other way around.

The world would be a much better place if more people realized this.

Re: Images that fool computer vision raise security concerns

#14

Post by userbinator »

compbio wrote: post #10

It is good to know that they need access to a lot of predictions from a net, before they can create an image that will "fool" the net, but look alien to humans. Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets (…

Furthermore, I am seeing the security concerns, but I figure this is far from a practical attack

Perhaps it's a sign of the times that almost every discovery that could possibly be related to security in some way, does. I have a feeling that if this was a decade or two ago, the sentiment would be very different. ("Can you figure out what a computer thinks these images are?")

Also, the image labeled "baseball" immediately reminded me of a baseball...

Re: Images that fool computer vision raise security concerns

#15

Post by pavel_lishin »

compbio wrote: post #10

It is good to know that they need access to a lot of predictions from a net, before they can create an image that will "fool" the net, but look alien to humans. Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets (…

> Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets

Sure, but this assumes that whatever neural net system you're relying on was bought by someone who is more security conscious than they are cheap.

Re: Images that fool computer vision raise security concerns

#16

Post by compbio »

jahnu wrote: post #2

> But computers don’t process images the way humans do, Yosinski said. This means that come the singularity AIs will have to use AI specific CAPTCHAs in order to distinguish between humans (aided by dumb computers) and other AIs.

If we have a singularity AI then we don't need CAPTCHAs: For all intents and purposes an AI is equal or superior to humans, there is no distinction to be made. The Turing Test is a form of CAPTCHA.

Also, one could make CAPTCHA's incredibly hard. Humans and dumb computers won't be able to solve it, and singular AI's get a pass. So give access to anyone who isn't able to solve the CAPTCHA and redirect the singularity AIs to google.com :).

Finally, the singular AI could create a CAPTCHA which separates humans from AI. The problem of separating humans from AI will quickly become too difficult for humans to solve.

Re: Images that fool computer vision raise security concerns

#17

Post by jerf »

compbio wrote: post #10

It is good to know that they need access to a lot of predictions from a net, before they can create an image that will "fool" the net, but look alien to humans. Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets (…

"Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images."

Trivially "solved" by treating the ensemble as a single object, then constructing a counterexample. My intuition suggests that while the resulting "fooled you" image may very slowly converge on something human recoginizable, it won't do so at a computationally-useful rate.

Re: Images that fool computer vision raise security concerns

#18

Post by higherpurpose »

z5h wrote: post #7

This raises an interesting question. Do we want computers to see "correctly", or to see how we see? Would a preferred computer vision system experience the Checker shadow illusion? http://en.wikipedia.org/wiki/Checker_shadow_illusion If yes, computer vision will be as fallible as ours. If no, then there will always be examples, like the ones presented, where computers will see something different than humans.

Does it matter if we want them to do it or not? NSA and China will build them to do that anyway.

Re: Images that fool computer vision raise security concerns

#19

Post by anon4 »

msandford wrote: post #8
pavel_lishin wrote: post #6

Earlier quoted context omitted.

Which of the following would you most prefer? A: a puppy, B: a pretty flower from your sweetie, or C: a large properly formatted data file?

Welp, it's official. I'm a computer.

No, you're the maintenance guy. Computers don't care about formatting. The correct answer is A, because "A" is drawn with just three straight lines and computers like straight lines.
Post reply on HN