Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

141–148 of 148 posts

Re: How Adversarial Attacks Work

#141

Post by j2kun »

LolWolf wrote: post #60
j2kun wrote: post #10

Earlier quoted context omitted.

> No [machine learning] system is 100% secure. That is what's being claimed without proof, when they meant "existing neural network systems are known to be insecure"

Sure, you can't claim that every machine learning system is susceptible to this attack (here's one that isn't, a model that always returns the same class on all inputs). But (apart from obvious trivialities) I suspect this is a more general problem with interesting properties—the claim the GP is making boils down to something like: a non-trivial[0] machine learning model with large parameter space is, with vanishing…

Unfortunately, it is known that many classes of problems can be learned with adversarial manipulation of an epsilon-fraction of the training data. See for example the paper "PAC learning with nasty noise," though there are many other adversarial noise models in which resilience can be proven (though proving it even for a specific concept class would be enough to prove it wrong that "all machine learning systems are vulnerable").

Re: How Adversarial Attacks Work

#142

Post by yters »

red75prime wrote: post #110
yters wrote: post #108

Earlier quoted context omitted.

Evidence that humans are fooled by adversarial examples? Optical illusions and other perception issues don't count, as explained by many others in this set of threads.

Camouflage. Paintings perceived as real 3d objects for a fraction of second.

You think these are equivalent to changing a single pixel and causing a complete misperception of the object which is believed with almost 100% certainty?

Re: How Adversarial Attacks Work

#143

Post by candiodari »

ChuckMcM wrote: post #34

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you…

I have heard from the police in many places that humans have the same problem, and they have it in practice:

There are places in almost every police zone where people driving misinterpret the situation almost perfectly consistently. Which means in some cases that the exact same accident keeps happening with a certain regularity, mostly just involving cars (and generally in that case nothing's done about it), but sometimes involving bikers or pedestrians and in those cases usually after a while the roads are changed so it stops happening.

So here we are:

1) there are "adversarial attacks" on the human mind, in existing road situations

2) those attacks were built into our road network, by humans, and presumably accidentally

3) they systematically lead to accidents and cause deaths, mostly people who weren't even the human misinterpreting the situation. Also, a lot of economic damage is caused.

4) we do one of two things to fix it:

a) we just accept this as a fact of life and don't care

b) we change the road situation until we purely accidentally hit one that doesn't get misinterpreted.

So how to deal with this for AI drivers ... I know ! How about the exact same way ?

AI drivers, I've been stuck behind them in MTV traffic enough to know this, are far safer than good human drivers. They far exceed the ability average human drivers. And they wipe the floor with the very best humans where it comes to patience with fellow road users.

Why do we hold them to a 100% standard ? Nobody and nothing, human or otherwise, matches up to a 100% perfect standard. A stick you use to beat a dog will not have a 100% success rate, once every 10 years or so that stick will break and maybe even injure the guy holding the stick by bouncing around, and yet somehow that is acceptable ...

We need to talk about how good drivers need to be. 100% is simply not an acceptable answer.

Re: How Adversarial Attacks Work

#144

Post by LolWolf »

comex wrote: post #105
LolWolf wrote: post #60

Earlier quoted context omitted.

Sure, you can't claim that every machine learning system is susceptible to this attack (here's one that isn't, a model that always returns the same class on all inputs). But (apart from obvious trivialities) I suspect this is a more general problem with interesting properties—the claim the GP is making boils down to something like: a non-trivial[0] machine learning model with large parameter space is, with vanishing…

That’s a very strong claim, though, given that we’re explicitly talking about hypothetical future systems as opposed to “existing neural network systems”. After all, taking the definition of “machine learning” to the extreme, it could include a system based on simulating an entire human brain. So the statement would necessarily imply that either: (a) brain simulation is fundamentally impossible for some reason (inser…

Not only do I think (b) is more likely than (a), I think (b) is strictly true. I mean, our concentration is quite limited and there are optical illusions which are, perhaps even more strongly than eps-adversarial examples for a given architecture, very general problems. So, given a "brain-simulated ML" thing (whatever that may mean, but let's assume it's the case), I suspect there may be a way of coming up with even more striking adversarial examples—but, considering that architectures vary heavily between humans, I suspect this would be quite person-specific.

Anyways, my final point is in saying that such a claim doesn't seem too far-fetched in any way.

That being said: I could be totally and hilariously wrong.

Re: How Adversarial Attacks Work

#145

Post by LolWolf »

j2kun wrote: post #141
LolWolf wrote: post #60

Earlier quoted context omitted.

Sure, you can't claim that every machine learning system is susceptible to this attack (here's one that isn't, a model that always returns the same class on all inputs). But (apart from obvious trivialities) I suspect this is a more general problem with interesting properties—the claim the GP is making boils down to something like: a non-trivial[0] machine learning model with large parameter space is, with vanishing…

Unfortunately, it is known that many classes of problems can be learned with adversarial manipulation of an epsilon-fraction of the training data. See for example the paper "PAC learning with nasty noise," though there are many other adversarial noise models in which resilience can be proven (though proving it even for a specific concept class would be enough to prove it wrong that "all machine learning systems are v…

But the claim the paper is making fits perfectly with the above:

> On the negative side, we prove that no algorithm can achieve accuracy of ε Though this is for a very particular type of adversary which has a lot more power than I'm claiming. In my case, though, I'm strengthening the side that the adversary has a very large number (e.g. infinite, as my claim "N->infty") of possible examples in the training set to be eps-close to.

---

Rereading the above: Sorry, I'm not quite sure I understood your point! Are you claiming that there is such a model which is resilient to adversarial examples, and also has a large number of parameters relative to the hypothesis class size? (e.g. say VC dimension?) In that case, my claim would be definitely false, but I have yet to see such a paper (if you have a reference, I'd love to check it out!)

Re: How Adversarial Attacks Work

#146

Post by red75prime »

yters wrote: post #142
red75prime wrote: post #110

Earlier quoted context omitted.

Camouflage. Paintings perceived as real 3d objects for a fraction of second.

You think these are equivalent to changing a single pixel and causing a complete misperception of the object which is believed with almost 100% certainty?

Single pixel attack is an artifact of the network architecture. It is not inherent vulnerability of every ML technique.

Re: How Adversarial Attacks Work

#147

Post by munificent »

romaniv wrote: post #66
munificent wrote: post #58

Earlier quoted context omitted.

> It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like that. Here's a physical object that makes you mistake an insect for a plant: https://en.wikipedia.org/wiki/Phasmatodea

Problem is, mimicry involves copying essential properties of the target object: color, texture, shape, movement dynamics and so on. It relies on true ambiguity. Adversarial examples against neural networks (the interesting ones, anyway) involve a combination of insignificant, seemingly random permutations that only work in their totality. That's a very important difference.

> essential properties of the target object

They are only "essential" properties according to the limitations of your particular perceptual system. To a bee that can see UV light, a stick bug may look entirely different from an actual stick. An animal that hunts by scent would find them clearly distinct.

There is no such thing as "true" ambiguity unless the two objects are actually the same thing in all respects. If they are distinct but appear the same, it's because they are overlapping in some respects but not all.

Re: How Adversarial Attacks Work

#148

Post by romaniv »

munificent wrote: post #147
romaniv wrote: post #66

Earlier quoted context omitted.

Problem is, mimicry involves copying essential properties of the target object: color, texture, shape, movement dynamics and so on. It relies on true ambiguity. Adversarial examples against neural networks (the interesting ones, anyway) involve a combination of insignificant, seemingly random permutations that only work in their totality. That's a very important difference.

> essential properties of the target object They are only "essential" properties according to the limitations of your particular perceptual system. To a bee that can see UV light, a stick bug may look entirely different from an actual stick. An animal that hunts by scent would find them clearly distinct. There is no such thing as "true" ambiguity unless the two objects are actually the same thing in all respects. If…

> They are only "essential" properties according to the limitations of your particular perceptual system. To a bee that can see UV light, a stick bug may look entirely different from an actual stick. An animal that hunts by scent would find them clearly distinct.

This is a very confused statement. Fist, since we are talking about image recognition we are - by definition - talking about vision in the spectrum that can be captured by a digital camera and encoded in a typical image format. Second, there definitely is such a thing as essential property. It is a matter of correlation with reality, as well as internal consistency. For example, plants are green and have leaves because of the way they use sunlight. So permuting color of all leaves in the picture is fundamentally different from permuting luminosity of some random pixels.

Post reply on HN