Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

41–50 of 148 posts

Re: How Adversarial Attacks Work

#41

Post by yters »

j2kun wrote: post #12
yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

Take care, the original authors were being overly broad when they implied "all ML algorithms" can be fooled in this way. Not that I disagree with your conclusion. Just the hypothesis.

All ML algorithms are either inconsistent or incomplete, which we know a priori. Insofar as ML tries to be complete, it will be inconsistent, as we see with adversarial examples.

Re: How Adversarial Attacks Work

#42

Post by ronald_raygun »

scrooched_moose wrote: post #28

Is there any reason to think this would work at all in the real world? All of these "attacks" require complete control of the image being fed to the classifier. In the ATM example you don't directly load an image of the check to the computer inside the machine. You design a check in photoshop, add the noise, print it out, feed it to the machine, which takes a picture of the check. Mobile bank apps still require you t…

Another example where they 3d printed a turtle to fool classifiers

https://twitter.com/daniel_bilar/status/925541876484231168

Re: How Adversarial Attacks Work

#43

Post by projectileboy »

ChuckMcM wrote: post #34

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you…

One possible mitigation could be the use of geocoded roadway metadata. I rented a car in France that had this - it would let me know when I was exceeding the speed limit based on my position. Not hyper-accurate, and so probably not a replacement for all road signs, but certainly good enough for many situations.

Re: How Adversarial Attacks Work

#44

Post by YeGoblynQueenne »

munificent wrote: post #17
yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only.

It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like that.

Note that I don't agree that this tells us anything about whether the human brain (or mind) is "like" a machine learning algorithm. To me this question has about as much meaning as asking if the brain is "like" quicksort.

The physical substrates are so clearly different that the only comparison you can make is on the level of capabilities (say, both are Turing-equivalent etc) not that of actual structures. Like, where's the 1's and 0's in the brain?

Re: How Adversarial Attacks Work

#45

Post by no_flags »

maltalex wrote: post #13
yters wrote: post #4

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

> this shows the human mind is not an ML algorithm. not necessarily. Humans have a lot more knowledge about contextual information that today's ML often misses. For every "image" you classify you use a LOT of side data. That side data has also been acquired through learning. Even if your actually saw a 1,000,000$ check or a 200 km/h speed limit, you'd know it's probably nonsense. Based on your life experience so far,…

What about the Sylvester Stallone/Keanu Reeves example? It was just a zoomed in head shot. That seems like an example where humans do a lot better without help from any additional context.

Re: How Adversarial Attacks Work

#46

Post by rtkwe »

squeaky-clean wrote: post #14
dontreact wrote: post #9

I don’t buy any of the attacks listed here or see how the examples being imperceptible is actually a factor. If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). If you are going to go out and replace speed limit signs to fool self driving cars, it’s probably equally dangerous whether or not the change is obvious, because if it’s way out of bo…

> If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). Plausible deniability? "I don't know why it took $1 million for a $100 check, look the check clearly says $100 on it, I didn't edit it. Must be a bank glitch."

All the deposit systems I've used recently still require you to enter the deposit amount so if you're entering 1000000 for a check that 'clearly' says 100 you lose all plausible deniability because you're actively lying to the bank at that point.

Re: How Adversarial Attacks Work

#47

Post by danharaj »

yters wrote: post #36
danharaj wrote: post #22

Earlier quoted context omitted.

Cool illusion; I figured it out and so will most humans. I don't think the claim that these sorts of illusions are of the same nature as, say, single pixel attacks against neural networks is justified. On the other hand, it is also true that it's not clear whether such attacks on our current crop of primitive neural networks will work on more advanced autonomous systems that regulate themselves.

In theory it seems that with sufficient understanding of any ML system, regardless of sophistication, such examples can be constructed. It is essentially the brain in the vat problem. Computers have no access to objective reality, whereas we cannot say the same regarding the human mind without making strong assumptions about the nature of the human mind. To put this in comp sci language, we can think of DNNs as proof…

I think the proper test is how stable the learning system is in the face of adversarial examples. I have no indications either way that we could, using current ideas, create a system that learns and is autonomous like a human brain while regulating itself from going too far off the rails. An optical illusion won't lead to you thinking you're the king of France, nor do there seem to be basilisks that will actually crash a brain.

It remains to be seen if the same holds for artificial learning systems. Of course! The gap between current learning systems and even the brain of a rodent is vast and we have yet no idea how to get there.

Re: How Adversarial Attacks Work

#49

Post by Winterflow3r »

naveen99 wrote: post #48

Don't you need access to the classifier internals to train the adversarial network ? Nobody is going to publish the network weights for a check reading machine...

No, this paper by Papernot et al shows how to do blackbox attacks without knowledge of model internals. https://arxiv.org/abs/1602.02697

Re: How Adversarial Attacks Work

#50

Post by c22 »

maltalex wrote: post #16
dontreact wrote: post #9

I don’t buy any of the attacks listed here or see how the examples being imperceptible is actually a factor. If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). If you are going to go out and replace speed limit signs to fool self driving cars, it’s probably equally dangerous whether or not the change is obvious, because if it’s way out of bo…

> If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). Humans are harder to fool. But some bank apps allow you to deposit a check by photographing it. Such apps would be fairly easy to attack. If you limit yourself to making 100$ checks become 1000$ checks and not 1,000,000$ checks, you might even get away with it.

A few years back a friend of mine was depositing a $300.00 check at Chase. When the teller asked him how she could help he said "Oh, just depositing this three thousand dollars." She punched in the deposit for $3000.00 and gave him a receipt. It was corrected within 24 hours but the receipt and printout of his bank statement was a great conversation piece. I doubt they would have let him just walk away with the money, nevermind multiple times.
Post reply on HN