Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keras reimplementation of "One pixel attack for fooling deep neural networks"

github.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#71

Post by avaxzat »

tomrod wrote: post #28
agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

Jumping on top comment (which I completely agree with) to ask: Why wouldn't a K-fold cross validation enable catching this? I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog). One could then attach a weighting, perhaps based on max RGB/CYMK vector norm betw…

>I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog).

While I can't speak for this attack in particular, there exist algorithms that can fool a neural network into generating high-confidence incorrect predictions for images that are visually indistinguishable from ones on which the network performs just fine. That's the biggest issue with these adversarial images.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#72

Post by avaxzat »

deegles wrote: post #15
xvedejas wrote: post #12

Earlier quoted context omitted.

The fact that a human isn't fooled by the attack (we can still recognize the 32x32 images for what they are), points to an interesting gap in the abilities of conventional convolutional neural nets.

That's only because the attack is designed to target that particular network. Just wait until we understand real brains better and can generate tailored attacks...

The one-pixel attack works for pretty much any machine learning model, not just one single neural network.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#73

Post by derivt »

This kind of attack relies on a low margin DNN, see (1), a low spectral norm of the input-output jacobian matrix guarantees good generalization error. So a one pixel attack exploits a weak eigenvalue (small absolute value) of the jacobian matrix.

So to create a one pixel attack, compute: 1)the eigenvalues of the jacobian of input-ouput matrix, 2) takes the the smaller eigenvalue lambda_1 3) compute or approximate the function lambda_1 = f(input) 4) compute j = argmax_{i=1..n} d(lambda_1)/d(input_i) at the point in which the spectral norm is maximum.

So to create the attack change the j-pixel in the points of the training set that has maximum (or high) jacobian matrix.

(1) https://arxiv.org/pdf/1605.08254.pdf

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#74

Post by mcintyre1994 »

Does this work if you use data transforms augmentation so that the pixel isn't always seen in the same spot? You probably wouldn't do any panning/rotation etc. for CIFAR 10, but for most practical purposes you probably would do more augmentation and I wonder if that defeats this?

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#75

Post by logicallee »

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

This is a truly excellent point. It applies to people, too: what will someone do if they are driving through the snow for the first time EVER, if they had lived in a temperate climate and just moved?

They will go very slow and perhaps it is not an exaggeration to say they will re-learn. Humans are learning constantly. In fact if there were some natural disaster (lava flow) and a human saw another car drive across some set lava on the way out of town (as more lava is rushing toward them) then a human will go ahead and follow, after seeing the other car make it through. If they see another car try to go across but get stuck on, they might take a detour and go find some intact bridge or other way to pass.

Actually, what you call "general" might be as much as general intelligence...

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#76

Post by smsm42 »

thaumasiotes wrote: post #55
smsm42 wrote: post #41

Earlier quoted context omitted.

Yes, and also this: https://en.wikipedia.org/wiki/Dazzle_camouflage Hacks human brain rather efficiently.

> Dazzle was adopted by the Admiralty in the UK, and then by the United States Navy, with little evaluation. Each ship's dazzle pattern was unique to avoid making classes of ships instantly recognisable to the enemy. The result was that a profusion of dazzle schemes was tried, and the evidence for their success was at best mixed. So many factors were involved that it was impossible to determine which were important,…

It is true that battlefield efficiency of such camouflage is unknown - but I think one can see the effects it does on the brain without conducting a proper rigorous study. The question here is not whether the effect exists - which is IMO obvious - but whether it's enough to make difference in actual combat.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#77

Post by tomrod »

yorwba wrote: post #51
tomrod wrote: post #40

Earlier quoted context omitted.

I must be misspeaking then. I'm not suggesting it run during training, but run the classification algorithm X number of times over an input image with chunks areas removed/suppressed from the data.

If I understand correctly, you want to modify an input image at application time to get multiple different classifications, and then compare them to find out how certain the model really is. While that would likely improve results a bit, it would also multiply the model runtime. That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the…

> That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty.

Exactly. It's a fix that doesn't work, apparently, so that's why I'm thinking towards the runtime.

> it would also multiply the model runtime.

Predictably so, I would think? Such an approach could scale decently since it's not adding a dimension to the runtime, just a multiple.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#78

Post by yorwba »

tomrod wrote: post #77
yorwba wrote: post #51

Earlier quoted context omitted.

If I understand correctly, you want to modify an input image at application time to get multiple different classifications, and then compare them to find out how certain the model really is. While that would likely improve results a bit, it would also multiply the model runtime. That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the…

> That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty. Exactly. It's a fix that doesn't work, apparently, so that's why I'm thinking towards the runtime. > it would also multiply the model runtime. Predictably so, I would think? Such an approach could scale decently since it's not adding a dimension to the runtime…

Predictably slow is still slow. You could parallelize it over multiple GPUs, but GPUs are expensive, so nobody is going to do that. Requiring lots of training resources is fine, but inference needs to be cheap.

More problematic is that your approach isn't going to actually work, since CNNs are just too flexible (they can learn even completely random labels) and only generalize by accident. No input augmentation technique that doesn't cover every possible modification is going to be robust against adversarial examples, and getting that amount of coverage requires an exponential blowup in runtime. The adversary has the advantage of being able to choose one modification, while the model needs to defend against all of them.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#80

Post by icc97 »

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

Shifting and mirroring is already a standard thing in papers with crop-10 (top-left, top-right, middle, bottom-left, bottom-right) * 2 (reflection on the horizontal axis). Colour shifting is also done via PCA analysis.

One thing that humans have is that young children watch the pages of a book turning, so see basic images at all extreme angles.

Post reply on HN