Hacking the Brain with Adversarial Images • Hacker News Board

Re: Hacking the Brain with Adversarial Images

#21

Post by mannykannot » Sat, Mar 03, 2018, 4:21 PM UTC

saulrh wrote: post #7

yters wrote: post #5

So if I make a cat look like a dog, people think it's a dog. That's amazing and totally unexpected.

The point is that the human brain is just as vulnerable to the adversarial attacks that people are claiming will be the downfall of ML. It's like the people that scream about people shining lasers into the cameras of autonomous vehicles - yeah, sure, if you did the same thing to a human driver you'd get exactly the same crash, why are you suddenly worried about this? And it's more than just "making it look like a dog…

The issue is not the risk of adversarial images being used to create mayhem, which was originally a straw man of the 'vehicle automation is infeasible' faction, but is becoming something of a straw man when it is used as the primary target of the 'this is not a problem' side. The issue is firstly the fragility of current image recognition in ways that are not easily bounded, and secondly the fact that humans use aspects of generalized intelligence in recognizing such situations and dealing with them, something that is not yet feasible to the same level in vehicle automation.

Re: Hacking the Brain with Adversarial Images

#22

Post by mikeash » Sat, Mar 03, 2018, 4:37 PM UTC

dgacmu wrote: post #4

matte_black wrote: post #2

With attacks like this it would never be safe to use image recognition in any applications where human life depends on accurate results.

Because my favorite hobby is modifying my liver so that the biopsy picture confuses the cancer detection algorithm. :)

You modify someone else’s liver, as a quick and easy method of assassination.

Re: Hacking the Brain with Adversarial Images

#23

Post by ccvannorman » Sat, Mar 03, 2018, 4:41 PM UTC

TLDR: If you modify pixels enough, you confuse humans, especially when you only let the humans look at the tiny blurry image for 70 ms.

rolls eyes I feel like this article totally misses the point about why adversarial ML tactics are interesting. The fundamental reason they work is that computers don't have any abstract notion of a cat, a toaster, a banana, or even gravity or matter. THAT is why they're easily fooled.

To stack up two adversarial images and say, "Look we fooled humans too, OMG humans are just like computers!" is like saying, "We erased the lines on the road and both self-driving cars AND humans got confused, OMG humans == computers!"

I know the article isn't saying humans == computers, but .. come on .. I'm not seeing any merit to this investigation

Re: Hacking the Brain with Adversarial Images

#24

Post by [deleted] » Sat, Mar 03, 2018, 4:52 PM UTC

[deleted]

Re: Hacking the Brain with Adversarial Images

#25

Post by your-nanny » Sat, Mar 03, 2018, 4:57 PM UTC

yters wrote: post #5

So if I make a cat look like a dog, people think it's a dog. That's amazing and totally unexpected.

So is blending an image of a dog and cat an adversarial modification of an image because the dog might be mistaken for the cat? It's just pixels. Some kind of definitional constraint on the sorts of modifications being made is prerequisite to this being actually interesting.

Re: Hacking the Brain with Adversarial Images

#26

Post by AnIdiotOnTheNet » Sat, Mar 03, 2018, 5:04 PM UTC

lowracle wrote: post #20

How far fetched would it be to assume that in the future, facebook would start building a neural network for each of its users, training it to like the same content as you ? With the amount of data generated by the endless scrolling, you could build a fairly accurate neural network of which content produce which emotion, and then generate adversarial images for targeted advertising.

At some point that neural network could just take over your social media presence and no one would ever know, because it's essentially your clone.

Re: Hacking the Brain with Adversarial Images

#27

Post by YeGoblynQueenne » Sat, Mar 03, 2018, 5:15 PM UTC

Woooah. Huuge assumptions here.

First of all, when we say that a classifier "misclassifies" an image, we have an agreed-upon definition of what that means: it means we have an image, we have a target label for it and the classifier assigns it a different label.

What exactly does it mean that a human misclassifies an image? More specifically, when the article says that "humans think that they're looking at something they aren't", what in the world does that mean? I mean, I have to assume a confusion between "looking", and "seeing". I can't really believe the article is saying that when I'm looking at the top right image I'm not actually, you know looking at it; but instead ...looking outside the window? Or what?

On the other hand- seeing? Really? Who knows what it is that anyone else is seeing? Who knows what is there to see? Especially when it comes to an image specifically manipulated to be confusing, as opposed to a real-world image? What is the ground truth here, for the image on the top-right? Is it a cat, just because Google reserachers say it's a cat and it's fooling my senses into thinking it's a dog ish? Is it fooling my senses just because the Google researchers say it is? Google researchers are the final arbiters of objective visible reality, now?

To be more precise, who is to say that that image can just be a "cat" or a "dog" and nothineg else? Because, you know, the first thing I thought when I saw that image on the top right was "that looks like a jumbled mess made with bits of an image of a cat and an image of a dog", or something along those lines.

Of course, if you sit me in front of a computer with two buttons, one for "dog" and one for "cat" like in the researchers' setup... well, then you can force me to misclassify the image all you like. But what does that prove? Besides the fact that if you force me to choose one of two things, without knowing what you think is the "right" thing to choose, I'll choose the "wrong" thing a lot of the time?

Re: Hacking the Brain with Adversarial Images

#28

Post by matte_black » Sat, Mar 03, 2018, 5:34 PM UTC

Filligree wrote: post #9

matte_black wrote: post #6

Earlier quoted context omitted.

I’m thinking more along the lines of cars, surveillance, heavy construction equipment etc.

https://xkcd.com/1958/

If this is the defense, it’s paper thin. Human drivers can easily reason about fake lanes, or fake pedestrian cut outs, or a sign somewhere that it shouldn’t be.

You also don’t need an abundance of murderers.

Re: Hacking the Brain with Adversarial Images

#29

Post by debatem1 » Sat, Mar 03, 2018, 5:34 PM UTC

Seems like an odd approach. The interesting thing about robust adversarial examples is that they fool computers but not humans. Something that fooled humans but not computers would seem to be the right next step. If you fool both it seems more likely that you just made an image that looks like what it classifies as.

Re: Hacking the Brain with Adversarial Images

#30

Post by ouid » Sat, Mar 03, 2018, 7:27 PM UTC

there's a lot of bullshit in this article.

To start, it should be clear to everyone that it is possible to actually transform a picture of a cat into a picture of a dog. The argument that the author is trying to make is not, in fact, that the human brain is hackable with adversarial images, but that the kinds of strategies which allow you to hack a wide range of deep learning models (to turn cats into dogs) require the introduction of "features" that a human would recognize as being doglike.

The leading image is not evidence of this, however. We have to keep track of the what figures correspond to which claims, because the article jumps around.

The first figure shows us a mask which people agree turns a cat into a dog. This is not evidence of any claim because such masks are guaranteed to exist.

The second figure shows a mask which people don't even detect which causes a computer to claim that a panda is a gibbon. This is basically the null hypothesis of the paper, it demonstrates that you can attack a single computer with a subtle mask.

The third figure shows us an example of an attack that is robust across deep learning models and also to perspective shifts. The argument being defended by this figure is that humans recognize the mistakes that the machine is making as being non-superficial features of laptops and toasters. The author undermines his own argument here, by requiring that the attack be robust to perspective shifts. The point he/she is trying to make is that in order to be robust to many neural nets you have to include non-superficial features, but then convolutes that claim by also requiring the features to be robust to other types of transformation as well. Does the "featuriness" of the attack come from the multiple neural network requirement or for the transformational robustness requirement? The answer is unknown.

Furthermore, I think this argument falls short pretty quickly if you simply hide the computer's answer. Would you guess that the computer guessed laptop for the first image, even if you knew it got it wrong? How far down your list would that have been? Or toaster for the second? The features offered in the second image definitely look like they have some features that are similar to the corner of a rounded metal cube, but this is exactly a superficial image statistic which is robust to change of perspective transformations. My classifier might as well be looking for that exactly. This is not evidence for the authors case.

The fourth and fifth figures are more interesting, but still do not provide any evidence in defense of machine learning.

It's important, for some reason, to point out that the accuracy on image tasks is NOT measured linearly with error rate. 99.9% accuracy is 10 times better than 99% accuracy. The difference between 65 and 75% is about 1000 times less than the difference between 75% and 99% Allow me to submit, without evidence, that humans can differentiate between a cat and a dog with >99% accuracy when given as much time as they need to make a decision about whether or not they are looking at a cat or a dog (with no adversarial mask applied). This means that the difference between human performance on the task without the time constraint, versus human performance on the task with the time constraint, is about 1000 times larger than the difference between human performance on the task with the time constraint and the task with the time constraint and the filter.

This means that if you really want people to fail on the cats versus dogs task, spending time coming up with the filter to apply after you've somehow managed to present the image that you're trying to get the person to misclassify for 1/20 of a second, followed by a noise mask, is >100 times less valuable at getting them to misclassify the picture than figuring out how to add the time constraint. Not to mention the fact that transformations that actually take cats to dogs and vice versa are necessarily possible, and that differentiation between cats and dogs is an image recognition task with a much clearer limit on human performance than differentiation between cats and laptops, which we are nevertheless amazingly good at.

Cats are very close to dogs, and it seems like this paper has completely forgotten this fact, and is deeply surprised to learn it.

Overall this is a very poor defense of deep learning.