Live data from Hacker News

Images that fool computer vision raise security concerns

news.cornell.edu

81–90 of 220 posts

Re: Images that fool computer vision raise security concerns

#81

Not directly related but, I was at a security related convention and overheard some people talking about an image that when occupying <3/4 of a frame will crash any digital camera. (Phone, DSLR, IP Camera) Does anyone know anymore information about this image and effect? I imagine it's a bug in some low level firmware of a common IC for digital photography DSP but, I'm very unfamiliar with digital cameras. It also co…

Could be related to this? http://en.wikipedia.org/wiki/EURion_constellation

Re: Images that fool computer vision raise security concerns

#82
post #7

This raises an interesting question. Do we want computers to see "correctly", or to see how we see? Would a preferred computer vision system experience the Checker shadow illusion? http://en.wikipedia.org/wiki/Checker_shadow_illusion If yes, computer vision will be as fallible as ours. If no, then there will always be examples, like the ones presented, where computers will see something different than humans.

Are all humans susceptible to the checker shadow illusion? Or just those that have been trained to interpret flat arrangements of color as accurate representations of 3D scenes and objects? If you showed the checker illusion to someone who had never seen a photograph or representative painting, would they see different colors or the same color?

I don't know the answer. I do find it fascinating that something as simple as perspective (which we take for granted in a graphic like the checkers illusion) is a fairly recent technique, invented by Renaissance artists.

Re: Images that fool computer vision raise security concerns

#83
post #74
post #27

Earlier quoted context omitted.

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…

Why should every crime go to trial? If the offender says "Fair cop, guv, you've got me bang to rights", why waste everyone's time proving that (s)he did it? It's basically the same as setting up your conditions to take advantage of short-circuit evaluation. You don't put the time-consuming and resource-hungry part first. Were there even the slimmest chance of acquittal, few defendants would utter that phrase without…

The issue isn't whether every case should go to trial. A plea bargain is an exchange of (forgoing) a trial for a reduced charge or sentence. The undesirable outcome (for pretty much everyone but the prosecutor) is that an innocent person accepts a plea bargain to avoid the worst outcome, and we should do more to prevent this from happening.

> If there's a suspicion of false confession, meaning that the suspect may not be the offender

Are you assuming that this represents a minority of cases? Who has to suspect that the accused is not really guilty, a jury of his peers?

Re: Images that fool computer vision raise security concerns

#85

They can only fool the DNN because they know their weights. In theory, you could do the same for people's brains. As long as you don't publish the specs of your net you should be fine I guess.

No, the images generated worked on even different neural networks trained on different data.

Re: Images that fool computer vision raise security concerns

#86
post #74
post #27

Earlier quoted context omitted.

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…

Why should every crime go to trial? If the offender says "Fair cop, guv, you've got me bang to rights", why waste everyone's time proving that (s)he did it? It's basically the same as setting up your conditions to take advantage of short-circuit evaluation. You don't put the time-consuming and resource-hungry part first. Were there even the slimmest chance of acquittal, few defendants would utter that phrase without…

>>Were there even the slimmest chance of acquittal, few defendants would utter that phrase without there being either a benefit to owning up, or an extra penalty for not doing so

Oh man, that's so wrong. Even where there is a LARGE change of acquittal, many choose a plea bargain because they cannot afford a good attorney or because the prosecutor is threatening them with something crazy like 40 years for downloading a movie. Would you take the risk of 40 years, knowing you're innocent? Do you have enough faith in a jury to put the rest of your life in their hands? I doubt it.

Re: Images that fool computer vision raise security concerns

#87
post #29
post #17

Earlier quoted context omitted.

"Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images." Trivially "solved" by treating the ensemble as a single object, then constructing a counterexample. My intuition suggests that while the resulting "fooled you" image may very slowly converge on something human recoginizabl…

Now I have to try this out. My intuition tells me it becomes increasingly hard to create a fooling image, which looks alien, and is able to fool all the nets in the ensemble, even though they have different settings and params. I think they can only fool one net at a time, and have to get very lucky to be able to evolve the image for the other nets, while keeping the same classification. You can't "train" these image…

Overfitting has nothing to do with it. See this paper: http://arxiv.org/abs/1412.6572

I believe the original paper tried ensembles and even got the images to work on different networks.

Re: Images that fool computer vision raise security concerns

#88
post #56

Earlier quoted context omitted.

> Basically neural networks and many other machine learning methods are highly linear No they're not! You introduce non-linearities like the sigmoid or tanh to make them highly non-linear.

I was thinking the same thing until I scanned through the paper linked above. While neural networks are indeed non-linear, some NNs can still exhibit what amounts to linearity and suffer from adversarial linear perturbations. Example of linearity in NNs that the authors are considering from the paper: >The linear view of adversarial examples suggests a fast way of generating them. We hypothesize that neural networks…

Right. The basic idea is something like the transition (manifolds) from doing special relativity to general relativity. The special "linear" says that given two inputs x and y to a function f, f is linear if f(x + y) = f(x) ⊕ f(y) for two operations +, ⊕. The general "linear" says that f(x + δx) = f(x) ⊕ δf(x, δx) for some small perturbations δx in the vicinity of x.

If x is a bit-vector then this can be as simple as saying "flip one bit of the input and here's how to predict which output bits get flipped." When you're building a hash function in cryptography, you try to push the algorithm towards a non-answer here: about half the bits should get flipped, and you shouldn't be able to predict which they are. But of course there's a security vulnerability even if + and ⊕ are not XORs.

Resisting "adversarial perturbation" in this context means basically that neural nets need to behave a bit more like hash functions, otherwise they will confuse the heck out of us. The problem is that if you just took the core lesson of hash functions -- create some sort of "round function" `r` so that the result is r(r(r(...r(x, 1)..., n - 2), n - 1), n) -- seems like it'd be really hard to invent learning algorithms to tune.

Re: Images that fool computer vision raise security concerns

#89

Earlier quoted context omitted.

Perhaps this is another way out of our problem? We could encourage police to "solve" property crimes rather than drug crimes? In analogy to their awful drug-war asset seizures, perhaps they could get a cut of items recovered/reimbursed? I'm sure they would eventually twist such an incentive structure into something else that's awful and unconscionable, but at least in the meantime they might stop with the SWAT raids?

Could we not just outlaw SWAT raids?

It's difficult (not impossible!) to directly oppose politically well-connected factions on their bread-and-butter issues. We might have more luck saying to police departments "here is another way you can have lots of money to spend" rather than "we're taking away federal support for the Drug War, which for some time has been the only way you can hire more cops and procure more equipment". The trick would be to make sure that the existing force is redirected into more benign directions, rather than simply growing to handle new activities while perpetuating the Drug War.

Re: Images that fool computer vision raise security concerns

#90
post #74
post #27

Earlier quoted context omitted.

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…

Why should every crime go to trial? If the offender says "Fair cop, guv, you've got me bang to rights", why waste everyone's time proving that (s)he did it? It's basically the same as setting up your conditions to take advantage of short-circuit evaluation. You don't put the time-consuming and resource-hungry part first. Were there even the slimmest chance of acquittal, few defendants would utter that phrase without…

I believe his point isn't that every crime necessarily needs to go to trial, it's that we should have enough resources to be able to put every accused criminal to trial. The fact that it's not the case shows the significant disconnect in our legal system.

That said, I don't believe it's very difficult to make an ethical and legal case against the concept of plea bargaining in the first place. I would argue it's very important to a free society to waste everyone's time proving that an accused person is guilty, and that not doing so violates the sixth amendment and fundamental human rights.

Post reply on HN