Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

121–130 of 148 posts

Re: How Adversarial Attacks Work

#121

Post by ben_w »

azernik wrote: post #92
ChuckMcM wrote: post #34

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you…

Or, simply with better machine learning. > The border between “truth” and “false” is almost linear. The first cool thing we can derive from it is that when you follow the gradient, once you find the area where the predicted class changes, you can be fairly confident that the attack is successful. On the other hand, it tells us that the structure the decision function is far simpler that most researchers thought it to…

Habe you seen the Monroe-Einstein illusion? There is a continuous transition from one image to the other, depending on the angular size. This feels relevant to your point.

https://static.independent.co.uk/s3fs-public/styles/article_...

Re: How Adversarial Attacks Work

#122

Post by ben_w »

throwaway613834 wrote: post #52
zardo wrote: post #32

Earlier quoted context omitted.

>Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically. Not true, we just have different priors and are fooled in different ways. See stage magic, optical illusions, etc...

> Not true, we just have different priors and are fooled in different ways. See stage magic, optical illusions, etc... But we can often recognize them as such, which is important. Actually, on that note, is there work on making machines being able to recognize magic tricks?

> But we can often recognize them as such, which is importan

Can we? The occult, new age, and religious sections of bookstores suggests otherwise, as do paid horoscope readings, homeopathic medicines, un/lucky numbers (housing and lottery), and shell-game scams.

Re: How Adversarial Attacks Work

#124

Post by yorwba »

naveen99 wrote: post #118
haraldurt wrote: post #86

Earlier quoted context omitted.

Not necessarily. Adversarial examples have been shown to, for instance, be transferable across different networks with different hyperparameters (e.g., number of layers) trained on disjoint subsets of a training set [0, section 4.2]. There are more references from the paper linked by the OP. [0] https://arxiv.org/abs/1312.6199

Thanks. I wonder if adversarial training helps prevent overfitting too. Could you use adversarial training to beat alphago ?

You could not, because AlphaGo is not a classifier (so it isn't well-defined what an adversarial example is) and the input space is discrete (Go board state) and you can't do ε-small perturbations (two different states differ by at least one stone).

Re: How Adversarial Attacks Work

#125

Post by tambourine_man »

ben_w wrote: post #121
azernik wrote: post #92

Earlier quoted context omitted.

Or, simply with better machine learning. > The border between “truth” and “false” is almost linear. The first cool thing we can derive from it is that when you follow the gradient, once you find the area where the predicted class changes, you can be fairly confident that the attack is successful. On the other hand, it tells us that the structure the decision function is far simpler that most researchers thought it to…

Habe you seen the Monroe-Einstein illusion? There is a continuous transition from one image to the other, depending on the angular size. This feels relevant to your point. https://static.independent.co.uk/s3fs-public/styles/article_...

I wouldn't call that an illusion, but more a consequence of a high-pass/low-pass filter at different distances with limited resolution.

What I find much more disturbing is that even though I measured and known those lines are parallel or that those shades of gray are the same, I can't "unsee" the illusion.

Re: How Adversarial Attacks Work

#127

Post by acallahan »

Obscurity seems like useful security here. IIUC it shouldn't be possible to e.g. trick self-driving cars with noisy signs, unless you have a copy of the classifier to train against. Thinking about ATMs, you could train against it as a black box, repeatedly inserting different patterns of noise? But it seems probably infeasible if you need to do a lot of iterations.

It also suggests that people concerned about adversarial attacks shouldn't use off-the-shelf pretrained classifiers, where attacks can be trained offline in advance. Similar to hashing algorithms and rainbow tables, maybe a practice of "salting" an off-the-shelf classifier could be effective in dodging attacks.

Re: How Adversarial Attacks Work

#128

Post by soundwave106 »

ben_w wrote: post #122
throwaway613834 wrote: post #52

Earlier quoted context omitted.

> Not true, we just have different priors and are fooled in different ways. See stage magic, optical illusions, etc... But we can often recognize them as such, which is important. Actually, on that note, is there work on making machines being able to recognize magic tricks?

> But we can often recognize them as such, which is importan Can we? The occult, new age, and religious sections of bookstores suggests otherwise, as do paid horoscope readings, homeopathic medicines, un/lucky numbers (housing and lottery), and shell-game scams.

All of those (along with stage magic and illusions) have plenty of material (in bookstores and elsewhere) describing either the mechanics, the long odds, or debunking them largely as pseudoscience and/or scams. So it's clear that some people at least can recognize them.

I think it's also possible for humans to be 100% aware of the "adversarial attack", and still use these types of mediums for light entertainment. This seems to describe many people who occasionally buy lottery tickets for entertainment, and would probably apply to many who attend and produce modern stage magic shows.

(In fact, I notice that some of the top modern crusaders against con artists who do use illusions and paranormal / occult claims for adversarial reasons are stage magicians themselves -- James Randi, Penn and Teller, Derren Brown.)

Re: How Adversarial Attacks Work

#129

Post by 3pt14159 »

acallahan wrote: post #127

Obscurity seems like useful security here. IIUC it shouldn't be possible to e.g. trick self-driving cars with noisy signs, unless you have a copy of the classifier to train against. Thinking about ATMs, you could train against it as a black box, repeatedly inserting different patterns of noise? But it seems probably infeasible if you need to do a lot of iterations. It also suggests that people concerned about adversa…

True. Also, just having unconnected systems that use different types of features / heuristics should be enough to at least pull the car over when they wildly disagree over what to do.

Re: How Adversarial Attacks Work

#130

Post by ChuckMcM »

stonewhite wrote: post #114
ChuckMcM wrote: post #81

Earlier quoted context omitted.

And they have: http://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-y... It just makes it harder to catch who did it.

Not relevant to the discussion, but turns out these kids didn't steal those signs. https://www.law.umich.edu/special/exoneration/Pages/casedeta...

Interesting to read that follow up. The kids had stolen 21 signs, but not those. Thanks for the link.
Post reply on HN