Keras reimplementation of "One pixel attack for fooling deep neural networks" • Hacker News Board

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#81

Post by icc97 » Sun, Feb 25, 2018, 12:13 PM UTC

But does this mean that edge detection is also failing?

If the neural network thinks a truck is a frog is it not recognising the vertical edges?

Seeing the intermediate layer images would be interesting to see where in the process it failed.

I keep thinking how kids often learn through labelled cartoon images. There the outline is more important.

Perhaps we could pre-train networks first on outlines of images. Make sure that these are capable of handling adversarial techniques and then build from there.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#82

Post by hyperparticle » Sun, Feb 25, 2018, 5:00 PM UTC

hyperparticle wrote: post #69

dahart wrote: post #21

Earlier quoted context omitted.

I suspect the same, that this kind of attack won't work on capnets. Note that "test on capnets" is on the list of not-yet-complete milestones.

Author of the repo here. I'm training a capsule network right now, will report back on results in the README in 24-48 hours. My hypothesis is that capsule networks are vulnerable as well, but we'll see by how much.

So I did some preliminary testing with CapsNet, and it seems that while it was harder than all the other CNNs to find adversarial pixels to fool the network, it was still vulnerable to attack. See the README for quantitative results.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#83

Post by eanzenberg » Sun, Feb 25, 2018, 7:33 PM UTC

madez wrote: post #63

eanzenberg wrote: post #54

Earlier quoted context omitted.

I’m not sure adding noise to the inputs of an equally complex model will change the information load of the NN. Because of the compression of the NN I think there will still exists new input pertubations which generate attacks.

My idea is that without added noise, knowing the model and the input allows crafting a pertubation that leads to an erroneous output of the model. With noise added there is less correlation between the input and the output. At the extreme with 100% randomness added, there is no correlation anymore between any pertubations of the input and the output. However, there is unfortunately also no correlation anymore between…

Adding noise is important for generalization and predictive power but it may just shift the attacking pixel from one to another, since the model itself is inherently compressed, inversely correlated with its complexity.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#84

Post by s3 » Wed, Feb 28, 2018, 12:14 AM UTC

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

Ian Goodfellow and others have done a lot of great work exploring the properties of adversarial attacks. While all the things you’ve mentioned are great practices and important to avoid overfitting, adversarial attacks are not due to overfitting (nor are they random) and none of the things you described are sufficient as defenses.