Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keras reimplementation of "One pixel attack for fooling deep neural networks"

github.com

Post reply on HN
Print view View original

21–30 of 84 posts

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#21

Post by dahart »

xvedejas wrote: post #13

I wonder how well capsule networks could deal with this, considering they're more robust than traditional CNNs towards other sorts of adversarial attacks. My guess is that one-pixel changes are going to do very little to alter pose information (though they will still alter the confidence of the feature existing), and so caps nets should be more robust here as well. Does anyone know if my intuition is correct? Link to…

I suspect the same, that this kind of attack won't work on capnets. Note that "test on capnets" is on the list of not-yet-complete milestones.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#22

Post by [deleted] »

tomhallett wrote: post #10
eximius wrote: post #3

Earlier quoted context omitted.

I mean, you don't even need a proof of that. The latter is impossible since the range of input/output values is untruncated by construction?

I'm very new to ML, so I understand about 50% of what @eximius and @goldenkey are saying, but definetly not 100%. can anyone explain it in a bit more detail? (im assuming "entropy" is the key concept i need to put on my learning queue.) “approximating a high dimensional function by clamping the entropy of the formula, rather than truncating the range of input/output values” “not poke through the entropically deprived…

[deleted]

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#23

Post by sdenton4 »

xvedejas wrote: post #12
7dare wrote: post #11

Isn't the fact that it's one pixel of a 32*32 image relevant? I'd be more impressed to see a neural network be succesfully attacked by a single (or dozens) of pixels on a full-res image.

The fact that a human isn't fooled by the attack (we can still recognize the 32x32 images for what they are), points to an interesting gap in the abilities of conventional convolutional neural nets.

A better analogy would be stimulating an individual receptor on the retina, in an eye with only 32^2 such receptors. When we see these pictures, we've got a much larger set of inputs to work with.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#24

Post by agitator »

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts of lighting, weather, and angle conditions, the training data is augmented, which means it is manipulated randomly with color, saturation, contrast, blurring, shifting, mirroring, adding noise, etc. So adding random pixels, to potentially protect against dead pixels is also a great idea.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#25

Post by tomrod »

tomhallett wrote: post #10
eximius wrote: post #3

Earlier quoted context omitted.

I mean, you don't even need a proof of that. The latter is impossible since the range of input/output values is untruncated by construction?

I'm very new to ML, so I understand about 50% of what @eximius and @goldenkey are saying, but definetly not 100%. can anyone explain it in a bit more detail? (im assuming "entropy" is the key concept i need to put on my learning queue.) “approximating a high dimensional function by clamping the entropy of the formula, rather than truncating the range of input/output values” “not poke through the entropically deprived…

I presume you have a regression background. If so: "minimizing entropy" is the same paradigm as "minimizing the sum of squares", in a different context.

If not: reducing entropy means finding weights/coefficients in a supplied functional form that minimize some objective function applied to the problem.

Usually the jargon applies to Shannon entropy from signal theory, or some derivation thereof like transfer entropy.

Entropic estimates take a form similar to

$$ -\sum(j) {p(x_j) log(p(x_j))}$$

where j is the event space (e.g. heads or tails on a coin flip).

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#26

Post by bradjohnson »

7dare wrote: post #11

Isn't the fact that it's one pixel of a 32*32 image relevant? I'd be more impressed to see a neural network be succesfully attacked by a single (or dozens) of pixels on a full-res image.

In the sources of the single pixel paper I found a reference to a prior work which does this: http://www.shivakasiviswanathan.com/CVPR17W.pdf

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#27

Post by legohead »

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

Walk around in the city with a giant yellow square costume and observe the mayhem...

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#28

Post by tomrod »

agitator wrote: post #24

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts…

Jumping on top comment (which I completely agree with) to ask:

Why wouldn't a K-fold cross validation enable catching this? I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog). One could then attach a weighting, perhaps based on max RGB/CYMK vector norm between two pixels across the image, to the folds' difference in top two certitudes.

I don't know, something like that.

Re: Keras reimplementation of "One pixel attack for fooling deep neural networks"

#29

Post by eximius »

tomhallett wrote: post #10
eximius wrote: post #3

Earlier quoted context omitted.

I mean, you don't even need a proof of that. The latter is impossible since the range of input/output values is untruncated by construction?

I'm very new to ML, so I understand about 50% of what @eximius and @goldenkey are saying, but definetly not 100%. can anyone explain it in a bit more detail? (im assuming "entropy" is the key concept i need to put on my learning queue.) “approximating a high dimensional function by clamping the entropy of the formula, rather than truncating the range of input/output values” “not poke through the entropically deprived…

If I have a function f(x,y, z) = \in R^5 (i.e., f: R^3 => R^5) that is hard to compute, I can train a neural network to approximate it such that there are three input neurons x', y', z', some hidden layers doing the approximation, and 5 output neurons representing the approximation of the input.

By construction, the domain and codomain are not constrained. Both the original and our approximation using NN take any three real values and return any five real values.

Next, consider a sample of points from some function. I can perfectly fit those points using a polynomial of degree equal to the number of points by just setting f(x) = (x-y_1)(x-y_2)... If, however, I approximate the function by removing some degrees from the formula, I remove information (entropy) from the formula. It is no longer a perfect match, but it might be very close. Or, if the underlying distribution is of low dimensionality, it might still be an exact match (i.e., picking any number of points from a straight line doesn't mean you need a high degree polynomial to approximate it!).

Post reply on HN