Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

101–110 of 148 posts

Re: How Adversarial Attacks Work

#101

Post by red75prime »

yters wrote: post #100
red75prime wrote: post #99

Earlier quoted context omitted.

> the human mind is not ML Pooling layers in CNNs throw away lots of information about spatial co-occurrence of features, which leads to a possibility of adversarial images where adversarial features are scattered all over the place and so they don't significantly affect humans' visual processing. The conclusion should be "Human mind is not that kind of ML".

CNNs are the best we have. If they can't deal with adversarial examples, then what can?

Hinton's "capsule networks", probably. Or any other future architecture which preserves positional information better.

Re: How Adversarial Attacks Work

#102

Post by cscheid »

alexcnwy wrote: post #98
sandGorgon wrote: post #37

Can such methods be used to "fingerprint" proprietary datasets by tainting them ? For example, i want to make sure that my dataset is not stolen and used by someone else (Waymo?). So I taint it using an adversarial method and create a "canary test set" that will uniquely identify if my dataset has been used in some training.

I believe google maps has fake streets as a canary for their maps being scraped...

yup, in cartography these are known as trap streets

Re: How Adversarial Attacks Work

#103

Post by yters »

red75prime wrote: post #101
yters wrote: post #100

Earlier quoted context omitted.

CNNs are the best we have. If they can't deal with adversarial examples, then what can?

Hinton's "capsule networks", probably. Or any other future architecture which preserves positional information better.

It is a problem inherent in any ML method with large VC dimension. The model will overfit, and when it overfits, adversarial examples become possible.

The very nature of ML seems incapable of dealing with this dilemma. You either work within a very constrained domain (low VC) to ensure everything is covered, and thus are unable to fit complex datasets, or expand the domain (large VC) and thus overfit the datasets. It's the bias-variance tradeoff, and is insurmountable.

Re: How Adversarial Attacks Work

#104

Post by chiaro »

YeGoblynQueenne wrote: post #44
munificent wrote: post #17

Earlier quoted context omitted.

I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.

The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only. It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like tha…

A more accurate example in the non-2D realm: https://en.wikipedia.org/wiki/Pareidolia

Re: How Adversarial Attacks Work

#105

Post by comex »

LolWolf wrote: post #60
j2kun wrote: post #10

Earlier quoted context omitted.

> No [machine learning] system is 100% secure. That is what's being claimed without proof, when they meant "existing neural network systems are known to be insecure"

Sure, you can't claim that every machine learning system is susceptible to this attack (here's one that isn't, a model that always returns the same class on all inputs). But (apart from obvious trivialities) I suspect this is a more general problem with interesting properties—the claim the GP is making boils down to something like: a non-trivial[0] machine learning model with large parameter space is, with vanishing…

That’s a very strong claim, though, given that we’re explicitly talking about hypothetical future systems as opposed to “existing neural network systems”. After all, taking the definition of “machine learning” to the extreme, it could include a system based on simulating an entire human brain. So the statement would necessarily imply that either:

(a) brain simulation is fundamentally impossible for some reason (insert your favorite argument for why this should be the case), or

(b) epsilon adversarial examples exist that fool humans (we just haven’t found any).

Personally, I think (b) is more likely than (a), but it seems much more likely that neither is true: that the human visual system is resilient at least to some extent (...for some epsilon), and could hypothetically be simulated by a machine. It would stand to reason that building a resilient system doesn’t require simulating a human, either, or nearly as much computational power as that would take (given that emulation is inherently very inefficient, and that visual processing only takes up a part of the human brain and seems to be done pretty well by animals with simpler brains). But it might require much more computational power than we have today…

On the other hand, if you limit computational power to around today’s level, and consider only potential architectural changes, the statement seems more plausible to me.

Re: How Adversarial Attacks Work

#106

Post by red75prime »

yters wrote: post #103
red75prime wrote: post #101

Earlier quoted context omitted.

Hinton's "capsule networks", probably. Or any other future architecture which preserves positional information better.

It is a problem inherent in any ML method with large VC dimension. The model will overfit, and when it overfits, adversarial examples become possible. The very nature of ML seems incapable of dealing with this dilemma. You either work within a very constrained domain (low VC) to ensure everything is covered, and thus are unable to fit complex datasets, or expand the domain (large VC) and thus overfit the datasets. It…

Adversarial examples are no longer adversarial in any meaningful sense, when humans are fooled by them too. We don't need to surmount BV tradeoff, we need to make sure that ML models make the same tradeoffs as humans.

Re: How Adversarial Attacks Work

#107

Post by samat »

Do this attacks still work if you don’t control input image bit-by-bit, but rather feed it via digital camera? While using any digits camera for input in the real world there will be some blurrines and distortion. Is it possible to construct such a noise that 1) will survive camera input 2) will be undetectable with human eyes 3) will fool a ML algorithm?

I do believe you are not able to control input image bit-by-bit in many real-world scenarios.

Re: How Adversarial Attacks Work

#108

Post by yters »

yters wrote: post #103
red75prime wrote: post #101

Earlier quoted context omitted.

Hinton's "capsule networks", probably. Or any other future architecture which preserves positional information better.

It is a problem inherent in any ML method with large VC dimension. The model will overfit, and when it overfits, adversarial examples become possible. The very nature of ML seems incapable of dealing with this dilemma. You either work within a very constrained domain (low VC) to ensure everything is covered, and thus are unable to fit complex datasets, or expand the domain (large VC) and thus overfit the datasets. It…

Evidence that humans are fooled by adversarial examples? Optical illusions and other perception issues don't count, as explained by many others in this set of threads.

Re: How Adversarial Attacks Work

#109

Post by taejo »

samat wrote: post #107

Do this attacks still work if you don’t control input image bit-by-bit, but rather feed it via digital camera? While using any digits camera for input in the real world there will be some blurrines and distortion. Is it possible to construct such a noise that 1) will survive camera input 2) will be undetectable with human eyes 3) will fool a ML algorithm? I do believe you are not able to control input image bit-by-bi…

As has been posted elsewhere in this thread, you can 3D print objects that are consistently misclassified when viewed through a camera from any angle: http://www.labsix.org/physical-objects-that-fool-neural-nets...

Re: How Adversarial Attacks Work

#110

Post by red75prime »

yters wrote: post #108
yters wrote: post #103

Earlier quoted context omitted.

It is a problem inherent in any ML method with large VC dimension. The model will overfit, and when it overfits, adversarial examples become possible. The very nature of ML seems incapable of dealing with this dilemma. You either work within a very constrained domain (low VC) to ensure everything is covered, and thus are unable to fit complex datasets, or expand the domain (large VC) and thus overfit the datasets. It…

Evidence that humans are fooled by adversarial examples? Optical illusions and other perception issues don't count, as explained by many others in this set of threads.

Camouflage. Paintings perceived as real 3d objects for a fraction of second.
Post reply on HN