Keras reimplementation of "One pixel attack for fooling deep neural networks" • Hacker News Board

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#51

Post by yorwba » Sat, Feb 24, 2018, 5:41 AM UTC

tomrod wrote: post #40

nightcracker wrote: post #37

Earlier quoted context omitted.

I don't believe that many people are using K-fold cross validation at all with deep learning, as the computational overhead is massive.

I must be misspeaking then. I'm not suggesting it run during training, but run the classification algorithm X number of times over an input image with chunks areas removed/suppressed from the data.

If I understand correctly, you want to modify an input image at application time to get multiple different classifications, and then compare them to find out how certain the model really is.

While that would likely improve results a bit, it would also multiply the model runtime. That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty.

However, random augmentation can't fully protect against adversarial examples. The number of input variables is simply too large, and there are exponentially many directions in which they could be modified. Data augmentation can't cover all of them, and a single modification that confuses the model slightly can be amplified into an adversarial example that causes a total misclassification.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#52

Post by eanzenberg » Sat, Feb 24, 2018, 6:08 AM UTC

tomrod wrote: post #28

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

Jumping on top comment (which I completely agree with) to ask: Why wouldn't a K-fold cross validation enable catching this? I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog). One could then attach a weighting, perhaps based on max RGB/CYMK vector norm betw…

Because its too computationally expensive. Attacks like these are found when you have access to the model, take an inout image and change its pixel values and see when an anomoly occurs. It’s much faster scoring N images vs. building N models.

How do you combat it? Well that’s an open research question. IMO the most promising techniques make the system harder to attack (ensemble models, more complex models, randomizing the input slightly and dropping outliers) but its not a guarantee. Like in security, it would be great to verify a model is safe to 200 years of brute force search for attack, or whatever it may be.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#53

Post by eanzenberg » Sat, Feb 24, 2018, 6:12 AM UTC

xvedejas wrote: post #13

I wonder how well capsule networks could deal with this, considering they're more robust than traditional CNNs towards other sorts of adversarial attacks. My guess is that one-pixel changes are going to do very little to alter pose information (though they will still alter the confidence of the feature existing), and so caps nets should be more robust here as well. Does anyone know if my intuition is correct? Link to…

I’m not sure capsules will change the security much. We’re talking about pixel level attacks while capsules are trying to generalize at larger length scales (rotational/translational invariance).

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#54

Post by eanzenberg » Sat, Feb 24, 2018, 6:20 AM UTC

madez wrote: post #34

tomhallett wrote: post #10

Earlier quoted context omitted.

I'm very new to ML, so I understand about 50% of what @eximius and @goldenkey are saying, but definetly not 100%. can anyone explain it in a bit more detail? (im assuming "entropy" is the key concept i need to put on my learning queue.) “approximating a high dimensional function by clamping the entropy of the formula, rather than truncating the range of input/output values” “not poke through the entropically deprived…

The set of all mappings between an input set of N elements to some output set with M elements has M^N elements. If you wanted to be able to represent in some way any arbitrary mapping for given sets of input and output, then you would need at least log_2(M^N) = N x log_2(M) bits. In the case of an input set of 32x32 pixel images with 3 bytes per pixel (one for each channel) we have N = 2^8 x 2^8 x 2^8 x 2^5 x 2^5 = 2…

I’m not sure adding noise to the inputs of an equally complex model will change the information load of the NN. Because of the compression of the NN I think there will still exists new input pertubations which generate attacks.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#55

Post by thaumasiotes » Sat, Feb 24, 2018, 6:41 AM UTC

smsm42 wrote: post #41

lbearl wrote: post #19

Earlier quoted context omitted.

Isn't that basically what an optical illusion is?

Yes, and also this: https://en.wikipedia.org/wiki/Dazzle_camouflage Hacks human brain rather efficiently.

> Dazzle was adopted by the Admiralty in the UK, and then by the United States Navy, with little evaluation. Each ship's dazzle pattern was unique to avoid making classes of ships instantly recognisable to the enemy. The result was that a profusion of dazzle schemes was tried, and the evidence for their success was at best mixed. So many factors were involved that it was impossible to determine which were important, and whether any of the colour schemes were effective.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#56

Post by argonaut » Sat, Feb 24, 2018, 6:50 AM UTC

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

It also doesn't work for adversarial attacks. "Adversarial training" helps somewhat but is still very susceptible to adversarial attacks.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#57

Post by alexcnwy » Sat, Feb 24, 2018, 7:54 AM UTC

eanzenberg wrote: post #52

tomrod wrote: post #28

Earlier quoted context omitted.

Jumping on top comment (which I completely agree with) to ask: Why wouldn't a K-fold cross validation enable catching this? I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog). One could then attach a weighting, perhaps based on max RGB/CYMK vector norm betw…

Because its too computationally expensive. Attacks like these are found when you have access to the model, take an inout image and change its pixel values and see when an anomoly occurs. It’s much faster scoring N images vs. building N models. How do you combat it? Well that’s an open research question. IMO the most promising techniques make the system harder to attack (ensemble models, more complex models, randomizi…

What about applying dropout to the input image

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#58

Post by anonytrary » Sat, Feb 24, 2018, 8:21 AM UTC

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

It would be interesting to try and generate this kind of data from 3D models: entire intricate scenes, where the camera can go around and take thousands of images in various lightings, weathers and vantage points. ICBW, but I feel like this modeling may have been tried and didn't translate well to real-life at the time. I can't see why it wouldn't be possible with enough modeling and physics precision, though.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#59

Post by Jaxan » Sat, Feb 24, 2018, 8:24 AM UTC

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

You make it sound a bit too easy “just add some random colours”. But there are theoretical hardness results from the 90s on learning NN. It is so-called inherently unpredictable. These hardness results became a bit obsolete by big data. I believe that what we see happening with adversarial inputs is a shadow of such hardness results.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#60

Post by Jaxan » Sat, Feb 24, 2018, 8:28 AM UTC

7dare wrote: post #11

Isn't the fact that it's one pixel of a 32*32 image relevant? I'd be more impressed to see a neural network be succesfully attacked by a single (or dozens) of pixels on a full-res image.

There are attacks where all pixels are just slightly changed and the classification is completely wrong. Think of changing all pixels by just the least significant bit. Still these changes are invisible to (my) human eye. Basically, each distance function has atttacks with a very small distance.