Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keras reimplementation of "One pixel attack for fooling deep neural networks"

github.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#61

Post by scoot »

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

This made me realize that Waymo's training data from Google street view imaging and recaptcha based labeling, whilst noisy in many natural ways, is exclusively daytime data. Seems a glaring hole.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#62

Post by [deleted] »

eanzenberg wrote: post #54
madez wrote: post #34

Earlier quoted context omitted.

The set of all mappings between an input set of N elements to some output set with M elements has M^N elements. If you wanted to be able to represent in some way any arbitrary mapping for given sets of input and output, then you would need at least log_2(M^N) = N x log_2(M) bits. In the case of an input set of 32x32 pixel images with 3 bytes per pixel (one for each channel) we have N = 2^8 x 2^8 x 2^8 x 2^5 x 2^5 = 2…

I’m not sure adding noise to the inputs of an equally complex model will change the information load of the NN. Because of the compression of the NN I think there will still exists new input pertubations which generate attacks.

[deleted]

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#63

Post by madez »

eanzenberg wrote: post #54
madez wrote: post #34

Earlier quoted context omitted.

The set of all mappings between an input set of N elements to some output set with M elements has M^N elements. If you wanted to be able to represent in some way any arbitrary mapping for given sets of input and output, then you would need at least log_2(M^N) = N x log_2(M) bits. In the case of an input set of 32x32 pixel images with 3 bytes per pixel (one for each channel) we have N = 2^8 x 2^8 x 2^8 x 2^5 x 2^5 = 2…

I’m not sure adding noise to the inputs of an equally complex model will change the information load of the NN. Because of the compression of the NN I think there will still exists new input pertubations which generate attacks.

My idea is that without added noise, knowing the model and the input allows crafting a pertubation that leads to an erroneous output of the model.

With noise added there is less correlation between the input and the output. At the extreme with 100% randomness added, there is no correlation anymore between any pertubations of the input and the output. However, there is unfortunately also no correlation anymore between the input and the output.

What happens if you add a bit of noise? The more noise, the smaller the correlation between the perturbations and the output. At what point is the probability of a successful attack sufficiently small?

To clarify, I mean adding the noise not in the training phase or to the images itself, but at the input stage into the model. That way even the repeated input of the same image would result in different inputs to the model.

I'm not sure this type of protection is efficient and effective, but it's an idea.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#64

Post by derivt »

We need to define the derivative of a deep model. I mean a way to measure how a model change when we change one pixel in the training data. Since pixel -> feature -> margin, we need to define the derivative with respect to a natural parameter, the natural parameter of the model has to defined ad hoc for every application. Perhaps the natural parameter encodes an uninformative prior. The intuition is to use information theory to see how the discriminative power of the model change when the training data is perturbed. So we need to measure the derivative of the added information. Fisher information seems to be related to this.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#65

Post by derivt »

derivt wrote: post #64

We need to define the derivative of a deep model. I mean a way to measure how a model change when we change one pixel in the training data. Since pixel -> feature -> margin, we need to define the derivative with respect to a natural parameter, the natural parameter of the model has to defined ad hoc for every application. Perhaps the natural parameter encodes an uninformative prior. The intuition is to use informatio…

this paper https://openreview.net/forum?id=HJC2SzZCW suggest that sensivity is related to poor generalization power. To define derivative we need to use a natural parameter in such a way that it measures sensivity and also allow us to use methods from calculus and manifolds, such as parallel transport of features. How a DNN label a cat when is catching a rat.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#66

Post by derivt »

derivt wrote: post #65
derivt wrote: post #64

We need to define the derivative of a deep model. I mean a way to measure how a model change when we change one pixel in the training data. Since pixel -> feature -> margin, we need to define the derivative with respect to a natural parameter, the natural parameter of the model has to defined ad hoc for every application. Perhaps the natural parameter encodes an uninformative prior. The intuition is to use informatio…

this paper https://openreview.net/forum?id=HJC2SzZCW suggest that sensivity is related to poor generalization power. To define derivative we need to use a natural parameter in such a way that it measures sensivity and also allow us to use methods from calculus and manifolds, such as parallel transport of features. How a DNN label a cat when is catching a rat.

If in a DNN for label a cat we explore the group of movements of the animal cat (realistic movements available for a cat) we could relate the discriminative power of the DNN to the energy of the cat. The energy of the cat is related to the volume of the group of movements. A cat with zero energy has the identity group of movements (no movement), a hulk cat is able to alter many of her features, so a very power model is needed to identify a hulk cat. A person full of rage is able to change the color of her face, again energy alter training space. Sorry for using HN for thinking.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#67

Post by hyperparticle »

7dare wrote: post #11

Isn't the fact that it's one pixel of a 32*32 image relevant? I'd be more impressed to see a neural network be succesfully attacked by a single (or dozens) of pixels on a full-res image.

The authors of the paper have successfully conducted the attack on 227*227 images as well. The pixels are much harder to see with the human eye.

It seems that larger images increase the search space as a linear function of the dimensions. That is to say, it does take more time to find such pixels, but they are still relatively common.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#68

Post by avaxzat »

bno1 wrote: post #43
deegles wrote: post #15

Earlier quoted context omitted.

That's only because the attack is designed to target that particular network. Just wait until we understand real brains better and can generate tailored attacks...

500,000 years ago our eyes and vision were probably significantly worse than they are today. As our eyes evolved to capture the world better our brains also evolved to correct the errors from our eyes. On the other hand, we feed into our neural networks high quality images. It's true that they are low resolution but they don't contain noticeable noise or artifacts. The attack described here is a smart application of…

>I'm curious whether training the network by adding noise and other mutations to the set would make the network more resilient to this attacks. In other words, it's the training set or the network architecture that's vulnerable here?

This is called adversarial training and is currently the most popular technique for protecting neural networks against this type of attack. That being said, it doesn't work as well as one would hope: the adversarially trained models are usually still vulnerable to other attacks.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#69

Post by hyperparticle »

dahart wrote: post #21
xvedejas wrote: post #13

I wonder how well capsule networks could deal with this, considering they're more robust than traditional CNNs towards other sorts of adversarial attacks. My guess is that one-pixel changes are going to do very little to alter pose information (though they will still alter the confidence of the feature existing), and so caps nets should be more robust here as well. Does anyone know if my intuition is correct? Link to…

I suspect the same, that this kind of attack won't work on capnets. Note that "test on capnets" is on the list of not-yet-complete milestones.

Author of the repo here. I'm training a capsule network right now, will report back on results in the README in 24-48 hours. My hypothesis is that capsule networks are vulnerable as well, but we'll see by how much.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#70

Post by derivt »

derivt wrote: post #66
derivt wrote: post #65

Earlier quoted context omitted.

this paper https://openreview.net/forum?id=HJC2SzZCW suggest that sensivity is related to poor generalization power. To define derivative we need to use a natural parameter in such a way that it measures sensivity and also allow us to use methods from calculus and manifolds, such as parallel transport of features. How a DNN label a cat when is catching a rat.

If in a DNN for label a cat we explore the group of movements of the animal cat (realistic movements available for a cat) we could relate the discriminative power of the DNN to the energy of the cat. The energy of the cat is related to the volume of the group of movements. A cat with zero energy has the identity group of movements (no movement), a hulk cat is able to alter many of her features, so a very power model…

Given that DNN are deep, what a one pixel attack means is that one pixel change propagates through the map of features: one pixel => 0-level-feature change -> one 1-level feature change. So this attack relies in weak features that can easily propagate to next level of features. Hence, to defend against this attack the model should put a threshold on the ratio (sensitivity of features)/(number of pixels) and avoid features with high sensivity to easily propagate to the next level of the DNN. If features are not linearly related to input set, then correlation is not a measure of feature sensitivity and has nothing to say about the full DNN effect of such change in a pixel.
Post reply on HN