Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Adversarial Attacks Work

blog.ycombinator.com

Post reply on HN
Print view View original

131–140 of 148 posts

Re: How Adversarial Attacks Work

#131

Post by ovi256 »

sandGorgon wrote: post #37

Can such methods be used to "fingerprint" proprietary datasets by tainting them ? For example, i want to make sure that my dataset is not stolen and used by someone else (Waymo?). So I taint it using an adversarial method and create a "canary test set" that will uniquely identify if my dataset has been used in some training.

You don't even need an adversarial attack method to fingerprint or watermark a proprietary dataset. If the dataset consists of images, you could just watermark them, or mark them using steganography. The watermark would appear as random (non-targeted) noise, and would be mostly invisible to current classifiers. Please note that the noise employed by adversarial attacks is very different, it is highly targeted.

Re: How Adversarial Attacks Work

#132

Post by YeGoblynQueenne »

jpttsn wrote: post #116
YeGoblynQueenne wrote: post #73

Earlier quoted context omitted.

Again, this is an optical illusion that fools depth perception. It doesn't make you see the T-Rex as a tea-pot, say. Different things.

Fair point; it's not exactly the same mistake. If humans made those, this discussion would probably never happen. What I tried to rebut is your first, weaker assertion: > The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only. This makes me see a mo…

OK, but I did say "the (vast) majority".

Also, from what I can tell, this just happens to work better with a meaningful shape (the image of a T-Rex). It has something to do with how we confuse convex with concave shapes and the T-Rex's snout just happens to be very convenient to demonstrate this confusion (e.g., I think an image of a horse would work as well). I think the illusion would also work with an abstract shape- except it would be harder to find a good abstract shape to demonstrate it as clearly as with the T-Rex.

Re: How Adversarial Attacks Work

#133

Post by theptip »

azernik wrote: post #92
ChuckMcM wrote: post #34

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you…

Or, simply with better machine learning. > The border between “truth” and “false” is almost linear. The first cool thing we can derive from it is that when you follow the gradient, once you find the area where the predicted class changes, you can be fairly confident that the attack is successful. On the other hand, it tells us that the structure the decision function is far simpler that most researchers thought it to…

Agree, it seems to me that there's a sense in which this is a trivial issue; just throw more adversarial examples into the training set, and turn the crank until the classifier understands how to correctly handle those cases too.

There's also a sense in which this is extremely non-trivial; any agent (human or machine) can be subjected to adversarial attacks. They just look different right now, and our current systems are vulnerable to very simple ones.

It seems to me that improving an algorithm's resistance to adversarial attacks is much more feasible than improving a human's resistance to their own class of adversarial attacks.

Re: How Adversarial Attacks Work

#134

Post by rtkwe »

throwaway613834 wrote: post #53
rtkwe wrote: post #46

Earlier quoted context omitted.

All the deposit systems I've used recently still require you to enter the deposit amount so if you're entering 1000000 for a check that 'clearly' says 100 you lose all plausible deniability because you're actively lying to the bank at that point.

Huh, I'd always wondered why they ask you to enter the number. Thanks for pointing this out!

Yeah it's a combo of that and also just a check on the CV that reads the checks. Mainly it's the latter since it'll be really rare for people to maliciously enter an incorrect amount.

Re: How Adversarial Attacks Work

#135

Post by aidenn0 »

Houshalter wrote: post #76
aidenn0 wrote: post #72

ELI5: Why are adversarial attacks not preventable by adding unpredictable noise to untrusted inputs?

The attacks are resistant to noise, or at least can made to be so. If every single input is tweaked in exactly the right direction, noise won't undo that. Most inputs will still be pointing in the adversarial direction. The noise will move some inputs back to their original position, but others will be pushed even further into adversarial territory.

Thanks, that makes a lot of sense.

Re: How Adversarial Attacks Work

#136

Post by SNIPERBABY »

Houshalter wrote: post #76
aidenn0 wrote: post #72

ELI5: Why are adversarial attacks not preventable by adding unpredictable noise to untrusted inputs?

The attacks are resistant to noise, or at least can made to be so. If every single input is tweaked in exactly the right direction, noise won't undo that. Most inputs will still be pointing in the adversarial direction. The noise will move some inputs back to their original position, but others will be pushed even further into adversarial territory.

Good to know, thanks!

Re: How Adversarial Attacks Work

#137

Post by f137 »

smogcutter wrote: post #54
projectileboy wrote: post #43

Earlier quoted context omitted.

One possible mitigation could be the use of geocoded roadway metadata. I rented a car in France that had this - it would let me know when I was exceeding the speed limit based on my position. Not hyper-accurate, and so probably not a replacement for all road signs, but certainly good enough for many situations.

Was thinking of this too, but what happens when the map and "ground truth" disagree? The machine would need to be able to identify the rare circumstances when it should trust the map over its senses. If it's capable of that, this wouldn't be a problem in the first place.

What would you do if confused by the road signs? A car could 1) see what other drivers doing 2) ask them - and this is easies with v2v 3) ask some central authority owe the air

Re: How Adversarial Attacks Work

#138

Post by dTal »

SomeStupidPoint wrote: post #117
smogcutter wrote: post #54

Earlier quoted context omitted.

Was thinking of this too, but what happens when the map and "ground truth" disagree? The machine would need to be able to identify the rare circumstances when it should trust the map over its senses. If it's capable of that, this wouldn't be a problem in the first place.

When driving, that probably doesn't really matter: If the map and road sign disagree, just do what everyone around you is doing (ie, go with the flow) until you recover agreement. It's not perfect, but stretches out attacks to having to compromise long stretches of road and multiple vehicle types. If the cars for the next mile somewhere can find pace, the whole group can; similarly, if any kind of car can find pace,…

I am now picturing a mile-long phalanx of self-driving cars gaily careening off the end of a destroyed bridge because the map and road signs disagree and they're all "going with the flow". Or to put it another way, if all your friends drove off a bridge, would you?

Re: How Adversarial Attacks Work

#139

Post by ben_w »

soundwave106 wrote: post #128
ben_w wrote: post #122

Earlier quoted context omitted.

> But we can often recognize them as such, which is importan Can we? The occult, new age, and religious sections of bookstores suggests otherwise, as do paid horoscope readings, homeopathic medicines, un/lucky numbers (housing and lottery), and shell-game scams.

All of those (along with stage magic and illusions) have plenty of material (in bookstores and elsewhere) describing either the mechanics, the long odds, or debunking them largely as pseudoscience and/or scams. So it's clear that some people at least can recognize them. I think it's also possible for humans to be 100% aware of the "adversarial attack", and still use these types of mediums for light entertainment. Thi…

“Some” is not ”often”. If there are any illusions that affect all humans, then by definition we cannot give them as examples because nobody realises they exist.

Yes, it is possible for people to know lottery odds and still play for the excitement. This does not invalidate the claim that many play the lottery with the expectation of winning, nor that people choose numbers superstitiously.

Re: How Adversarial Attacks Work

#140

Post by j2kun »

shmel wrote: post #87
j2kun wrote: post #10

Earlier quoted context omitted.

> No [machine learning] system is 100% secure. That is what's being claimed without proof, when they meant "existing neural network systems are known to be insecure"

Really? I believe they meant exactly what they said. Read https://arxiv.org/abs/1605.07277 Adversarial examples can be easily crafted for linear regression, SVM, decision trees and k-NN at very least. I can't think about any ML technique that is proven to be secure against them. There is also an old blogpost from Karpathy describing that the only reason we are not concerned about adversarial examples in linear models…

Yes, many models are vulnerable. But there are no proofs that allow you to say that. You should also be aware that they're only talking about image classification in all of these.

I will only accept blanket statements about "all machine learning systems" when there's a mathematical proof.

I don't mean to lessen the importance of the work, just to point out that saying "X fails at Y on subtask Z using the techniques we have today" is very different from saying "All possible X's, current and future, fail at Y."

Post reply on HN