Not directly related but, I was at a security related convention and overheard some people talking about an image that when occupying <3/4 of a frame will crash any digital camera. (Phone, DSLR, IP Camera) Does anyone know anymore information about this image and effect? I imagine it's a bug in some low level firmware of a common IC for digital photography DSP but, I'm very unfamiliar with digital cameras. It also co…
Images that fool computer vision raise security concerns
81–90 of 220 posts
Re: Images that fool computer vision raise security concerns
#82This raises an interesting question. Do we want computers to see "correctly", or to see how we see? Would a preferred computer vision system experience the Checker shadow illusion? http://en.wikipedia.org/wiki/Checker_shadow_illusion If yes, computer vision will be as fallible as ours. If no, then there will always be examples, like the ones presented, where computers will see something different than humans.
I don't know the answer. I do find it fascinating that something as simple as perspective (which we take for granted in a graphic like the checkers illusion) is a fairly recent technique, invented by Renaissance artists.
Re: Images that fool computer vision raise security concerns
#83Earlier quoted context omitted.
This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…
Why should every crime go to trial? If the offender says "Fair cop, guv, you've got me bang to rights", why waste everyone's time proving that (s)he did it? It's basically the same as setting up your conditions to take advantage of short-circuit evaluation. You don't put the time-consuming and resource-hungry part first. Were there even the slimmest chance of acquittal, few defendants would utter that phrase without…
> If there's a suspicion of false confession, meaning that the suspect may not be the offender
Are you assuming that this represents a minority of cases? Who has to suspect that the accused is not really guilty, a jury of his peers?
Re: Images that fool computer vision raise security concerns
#84Machine pareidolia? On a serious not couldn't we keep training the same DNN's using these white noise images as negative examples?
Re: Images that fool computer vision raise security concerns
#85They can only fool the DNN because they know their weights. In theory, you could do the same for people's brains. As long as you don't publish the specs of your net you should be fine I guess.
Re: Images that fool computer vision raise security concerns
#86Earlier quoted context omitted.
This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…
Why should every crime go to trial? If the offender says "Fair cop, guv, you've got me bang to rights", why waste everyone's time proving that (s)he did it? It's basically the same as setting up your conditions to take advantage of short-circuit evaluation. You don't put the time-consuming and resource-hungry part first. Were there even the slimmest chance of acquittal, few defendants would utter that phrase without…
Oh man, that's so wrong. Even where there is a LARGE change of acquittal, many choose a plea bargain because they cannot afford a good attorney or because the prosecutor is threatening them with something crazy like 40 years for downloading a movie. Would you take the risk of 40 years, knowing you're innocent? Do you have enough faith in a jury to put the rest of your life in their hands? I doubt it.
Re: Images that fool computer vision raise security concerns
#87Earlier quoted context omitted.
"Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images." Trivially "solved" by treating the ensemble as a single object, then constructing a counterexample. My intuition suggests that while the resulting "fooled you" image may very slowly converge on something human recoginizabl…
Now I have to try this out. My intuition tells me it becomes increasingly hard to create a fooling image, which looks alien, and is able to fool all the nets in the ensemble, even though they have different settings and params. I think they can only fool one net at a time, and have to get very lucky to be able to evolve the image for the other nets, while keeping the same classification. You can't "train" these image…
I believe the original paper tried ensembles and even got the images to work on different networks.
Re: Images that fool computer vision raise security concerns
#88Earlier quoted context omitted.
> Basically neural networks and many other machine learning methods are highly linear No they're not! You introduce non-linearities like the sigmoid or tanh to make them highly non-linear.
I was thinking the same thing until I scanned through the paper linked above. While neural networks are indeed non-linear, some NNs can still exhibit what amounts to linearity and suffer from adversarial linear perturbations. Example of linearity in NNs that the authors are considering from the paper: >The linear view of adversarial examples suggests a fast way of generating them. We hypothesize that neural networks…
If x is a bit-vector then this can be as simple as saying "flip one bit of the input and here's how to predict which output bits get flipped." When you're building a hash function in cryptography, you try to push the algorithm towards a non-answer here: about half the bits should get flipped, and you shouldn't be able to predict which they are. But of course there's a security vulnerability even if + and ⊕ are not XORs.
Resisting "adversarial perturbation" in this context means basically that neural nets need to behave a bit more like hash functions, otherwise they will confuse the heck out of us. The problem is that if you just took the core lesson of hash functions -- create some sort of "round function" `r` so that the result is r(r(r(...r(x, 1)..., n - 2), n - 1), n) -- seems like it'd be really hard to invent learning algorithms to tune.
Re: Images that fool computer vision raise security concerns
#89Earlier quoted context omitted.
Perhaps this is another way out of our problem? We could encourage police to "solve" property crimes rather than drug crimes? In analogy to their awful drug-war asset seizures, perhaps they could get a cut of items recovered/reimbursed? I'm sure they would eventually twist such an incentive structure into something else that's awful and unconscionable, but at least in the meantime they might stop with the SWAT raids?
Could we not just outlaw SWAT raids?
Re: Images that fool computer vision raise security concerns
#90Earlier quoted context omitted.
This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…
Why should every crime go to trial? If the offender says "Fair cop, guv, you've got me bang to rights", why waste everyone's time proving that (s)he did it? It's basically the same as setting up your conditions to take advantage of short-circuit evaluation. You don't put the time-consuming and resource-hungry part first. Were there even the slimmest chance of acquittal, few defendants would utter that phrase without…
That said, I don't believe it's very difficult to make an ethical and legal case against the concept of plea bargaining in the first place. I would argue it's very important to a free society to waste everyone's time proving that an accused person is guilty, and that not doing so violates the sixth amendment and fundamental human rights.