Images that fool computer vision raise security concerns • Hacker News Board

Re: Images that fool computer vision raise security concerns

#101

Post by Houshalter » Mon, Mar 23, 2015, 6:00 PM UTC

compbio wrote: post #96

Houshalter wrote: post #87

Earlier quoted context omitted.

Overfitting has nothing to do with it. See this paper: http://arxiv.org/abs/1412.6572 I believe the original paper tried ensembles and even got the images to work on different networks.

I was not talking about overfitting. I've seen that paper. The original paper asked if images that could fool DBN.a could fool DBN.b. The answer was: certainly not all the time. They used the exact same train set and architecture for DBN.a and DBN.b, just randomly varied initial weights. I think this is too favorable for a comparison with a voting ensemble made with nets with a different architecture, train set and t…

To be clear I meant this paper (http://arxiv.org/abs/1312.6199) as the original paper for adversarial images. I think they did try transferring them between very different NNs:

>In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.

Re: Images that fool computer vision raise security concerns

#102

Post by JoeAltmaier » Mon, Mar 23, 2015, 6:02 PM UTC

sp332 wrote: post #91

JoeAltmaier wrote: post #37

Earlier quoted context omitted.

The OP shows that computers DO NOT have a better understanding. Its evident they have no understanding at all; they are simply doing math on pixels and latching on to coincidental patters of color or shading. People recognize things by building a 3D model in their head, then comparing that to billions of experiential models, finding a match and then using cognition to test that match. "Is that a bird? No, its just a…

I don't think your description of how humans recognize things is true. You can do object recognition of a silhouette, or a 2D image, or an impressionistic rendering. http://www.sciencedaily.com/releases/2009/04/090429132231.ht...

We can't help but build real models of what we see - our retina/optic nerve are already doing this before our brain even receives the 'image'!

I can't help but believe some of the image recognition mentioned in your article, especially of icons, is built through previous experience with similar iconic images. Symbols for things become associated with the real things. Its a modern adaptation of a much older processing mechanism.

Re: Images that fool computer vision raise security concerns

#103

Post by edison_carter » Mon, Mar 23, 2015, 6:03 PM UTC

mikeash wrote: post #27

toomuchtodo wrote: post #11

Earlier quoted context omitted.

> If everyone is breaking so many laws that the police and courts can't keep up it doesn't mean that humanity is broken. It means that the law has gotten so far out of sync with humanity that the law is broken. People make the laws, not the other way around. The world would be a much better place if more people realized this.

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…

You're right; there is tremendous pressure from both district attorneys and judges to cop a plea. Trials piss everyone off because they are "such a waste of time." Criminal defense attorneys have to contend not just with the facts of the case and legal precedents, but also the extent to which demanding a jury trial turns the court against your client -- and your other clients, by association.

The other issue is police overcharging crimes. For example, suppose you, in a fit of pique, cut your roommate's arm with a knife. Not cool; that's aggravated assault. But the police then charges you with attempted second degree murder -- or even first degree (eg, premeditated). Now your defense attorney has to work hard just to get your charges down to a reasonable level; the agg assault you should have been charged with in the first place. This is what a huge majority of plea bargaining is about; not getting away with it, but getting the charge down to something that describes your actual offense.

(Source: longtime criminal defense nerd)

Re: Images that fool computer vision raise security concerns

#104

Post by mikeash » Mon, Mar 23, 2015, 6:05 PM UTC

vinceguidry wrote: post #44

mikeash wrote: post #27

Earlier quoted context omitted.

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…

The problem is that you can't get very far into the discussion before all the available options involve overhauling the entire system. And there's no way, realistically, to do that.

Significantly reducing the number of criminals is fairly easy without overhauling the system. Decriminalize drug use and possession. That'll cut the number of trials (or pleas) substantially all by itself, and even more so if you assume that knock-on effects will reduce crime overall, as many drug legalization proponents think it would. Other nonviolent offenses could be cut back as well. Stop putting people in prison because one of them paid the other one for sex, for example. Here in Virginia, you can potentially go to jail for a year (and, obviously, have a trial if you want it) for driving at 80MPH in a zone marked for 70MPH!

Increasing resources for holding trials is also not terribly difficult and doesn't require any sort of overhaul. It's just a problem of money, and not a big one. Just for a random example, it looks like the courts account for about 0.3% of my local county budget and about 1% of my local state budget. We could literally increase court resources by a factor of 10 with only a modest increase in taxes to fund it.

Re: Images that fool computer vision raise security concerns

#105

Post by c-slice » Mon, Mar 23, 2015, 6:05 PM UTC

This isn't very groundbreaking, they're simply discovering the inherent weaknesses in neural net learning. We've know for years that neural nets have inherent gaps in their training and this is just taking advantage of that. This isn't really an issue in a real life scenario as you wouldn't be able to determine without thousands of recursions which images produce faulty results.

Re: Images that fool computer vision raise security concerns

#106

Post by mikeash » Mon, Mar 23, 2015, 6:06 PM UTC

happyscrappy wrote: post #54

mikeash wrote: post #27

Earlier quoted context omitted.

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…

> you're making too many criminals 80% of property crime goes unsolved, so unless you legalize vandalism and theft there are going to be criminals.

The 80% of property crime that goes unsolved is, by definition, not causing an excessive load on the court system.

Re: Images that fool computer vision raise security concerns

#107

Post by im2w1l » Mon, Mar 23, 2015, 6:09 PM UTC

lars wrote: post #52

Balgair wrote: post #34

Earlier quoted context omitted.

You can always fool any system into a paradox. Very grossly speaking, this comes out of Godel's incompleteness theorem; that with any set of laws you can always get P=~P out of the set. How this paradox looks, acts, or feels is interesting and possibly artistic, as the OP shows. If anything, I think there is a bit of beauty, art, and cleverness in that.

That isn't quite right. With a sufficiently powerful formal system, you're forced to either have inconsistency or incompleteness - you're describing a system that is inconsistent. It's usually much better to have consistency and to sacrifice completeness. Then you'll have Ps that are true but unprovable, but at least you won't have P=~P which makes the system rather useless.

What does true but unprovable mean? What happens if you take such a proposition, negate it and add as an axiom?

Re: Images that fool computer vision raise security concerns

#108

Post by scribu » Mon, Mar 23, 2015, 6:14 PM UTC

compbio wrote: post #10

It is good to know that they need access to a lot of predictions from a net, before they can create an image that will "fool" the net, but look alien to humans. Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets (…

> You won't be granted access to secret information if you happen to look like the CEO.

Tell that to this startup: http://onevisage.com/ovi-technology-page/

Re: Images that fool computer vision raise security concerns

#109

Post by compbio » Mon, Mar 23, 2015, 6:18 PM UTC

Houshalter wrote: post #101

compbio wrote: post #96

Earlier quoted context omitted.

I was not talking about overfitting. I've seen that paper. The original paper asked if images that could fool DBN.a could fool DBN.b. The answer was: certainly not all the time. They used the exact same train set and architecture for DBN.a and DBN.b, just randomly varied initial weights. I think this is too favorable for a comparison with a voting ensemble made with nets with a different architecture, train set and t…

To be clear I meant this paper ( http://arxiv.org/abs/1312.6199 ) as the original paper for adversarial images. I think they did try transferring them between very different NNs: >In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.

Very interesting. Thank you for correcting me.

a relatively large fraction of examples will be misclassified by networks trained from scratch with different hyper-parameters (number of layers, regularization or initial weights). The above observations suggest that adversarial examples are somewhat universal...

Re: Images that fool computer vision raise security concerns

#110

Post by wmil » Mon, Mar 23, 2015, 6:20 PM UTC

mikeash wrote: post #27

toomuchtodo wrote: post #11

Earlier quoted context omitted.

> If everyone is breaking so many laws that the police and courts can't keep up it doesn't mean that humanity is broken. It means that the law has gotten so far out of sync with humanity that the law is broken. People make the laws, not the other way around. The world would be a much better place if more people realized this.

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial. If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypa…

This is a big issue in the US and it actually goes back to the Warren court. They issued a long series of rulings making it difficult to prosecute cases, without worrying about the consequences.

By the 70s crime had skyrocketed and it was clear that they had gone too far. But instead of issuing a mea culpa and reexamining past rulings, the various courts started allowing prosecutors to claim broad new powers and take extremely aggressive tactics.

By this point there's no real way to fix it. The legal system is based strongly on precedent. It can't just undo major rulings of the past and replace the with something sane.