Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adversarial image attacks are no joke

unite.ai

Post reply on HN
Print view View original

121–130 of 196 posts

Re: Adversarial image attacks are no joke

#121

Post by hermitdev »

catlikesshrimp wrote: post #88
Vetch wrote: post #53

Earlier quoted context omitted.

Humans are not so bad as drivers. Your example is an event from over 2 decades ago and was deemed newsworthy. Humans drive in all kinds of conditions but death rate is about 1 per 100 million miles driven. A search reveals crashes to be on the order of hundreds of collisions per 100 million miles driven. Age, country, intoxication level, road design and laws, road and environmental conditions also play a major role s…

I will go further and say that almost everytime there is an accident the driver is somehow impaired. Lack of sleep, drugs, illness (old age included, mental disease), poor judgement (young age included, emotional distress) Humans are surprisingly good at driving under normal conditions.

Some are. Some are not. Last week, I was nearly in two accidents on maybe a 1 mile trip to the store from my house. Both times were people pulling out of traffic, ignoring right of way. I prevented the accidents that would have resulted from these two separate idiots. I have also been in over 20 accidents in my 25 years of driving, the vast majority of those having been rear-ended and none were my fault.

In my experience, I've not been in an accident with a teen, nor someone elderly, though I know people that have (both causing and being involved). Neither have I been in an accident with someone that I could tell was impaired by drugs or alcohol. I don't know for sure any of them involved a phone for that matter. Weather was only a factor in one accident (pouring rain, low visibility).

I have nothing to suggest that any of my accidents were caused by anything other than inattentiveness, even the one time weather played a minor role. I also see a lot of dangerous behavior every time I drive: people running lights and stop signs, completely ignoring yield signs (seriously, they must be invisible to everyone else), failing to yield right of way, failing to signal turns and lane changes (my favorite is turning the signal on after moving into the turn lane), lots of phone usage (for everything except making a call, from maps to texting to watching videos!).

Re: Adversarial image attacks are no joke

#122

Post by glitchc »

mabbo wrote: post #114
glitchc wrote: post #102

Earlier quoted context omitted.

Adversarial examples don't confuse people, only algorithms. Perhaps you need to face the fact that if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. I don't trust my life to your "It kinda looks like a road, oh wait it's a pylon, I've been tricked, BAM!" dumpster fire of an algorithm. We used to have to craft robustness i…

> if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. This is the tricky bit. Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental st…

> Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental stamina' can make a big difference in safe driving.

To be clear we are talking about CV which relies on passive optical sensing in the visual spectrum through cameras, not radar or lidar or IR or multi-spectral sensors.

Within this context, your statement is incorrect. A typical camera’s dynamic range is orders of magnitude lower than the human visual dynamic range. Ergo a camera sees a lot less at night compared to a human and what it does see is a lot more noisy. Note that this is the input to the detection, tracking and classification stages, the ouput of which feeds into the control loop(s). It doesn’t matter how good the control system is, it cannot avoid what the vision system cannot see.

Re: Adversarial image attacks are no joke

#123

Post by ummonk »

hermitdev wrote: post #121
catlikesshrimp wrote: post #88

Earlier quoted context omitted.

I will go further and say that almost everytime there is an accident the driver is somehow impaired. Lack of sleep, drugs, illness (old age included, mental disease), poor judgement (young age included, emotional distress) Humans are surprisingly good at driving under normal conditions.

Some are. Some are not. Last week, I was nearly in two accidents on maybe a 1 mile trip to the store from my house. Both times were people pulling out of traffic, ignoring right of way. I prevented the accidents that would have resulted from these two separate idiots. I have also been in over 20 accidents in my 25 years of driving, the vast majority of those having been rear-ended and none were my fault. In my experi…

> I have also been in over 20 accidents in my 25 years of driving, the vast majority of those having been rear-ended and none were my fault.

Do you just drive a lot or do you brake too late / too hard? Because an accident rate that high is rather unusual.

Re: Adversarial image attacks are no joke

#124

Post by yjftsjthsd-h »

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

> I believe the same will be true of adversarial examples for vision (and language) models. We can try to make them hard to make, hard to posses anonymously, and hard to deliver. I think this will be much easier with computer vision than with poison, so I'm not worried about it.

Erm. We can maybe do something about delivery, but stopping people from making (and thus, possessing) them is virtually impossible, since all you need is an undergrad-level understanding of ML (if that) and some freely-available software.

Re: Adversarial image attacks are no joke

#125

Post by p2p_astroturf »

Ever since I had the misfortune of learning about hacker kids wanting self-driving cars, I've been saying you can literally put a poster on the side of the road and every car that comes by it will crash. Seems like I'm on the right track. Software has edge cases. Every software engineer knows this.

>The second-most frequent complaint is that the adversarial image attack is ‘white box’, meaning that you would need direct access to the training environment or data.

The training data will be leaked. Companies are very bad at classifying what is and isn't private information that they need to keep secret. But anyway you probably don't even need the training data.

Re: Adversarial image attacks are no joke

#126

Post by frenchyatwork »

ummonk wrote: post #123
hermitdev wrote: post #121

Earlier quoted context omitted.

Some are. Some are not. Last week, I was nearly in two accidents on maybe a 1 mile trip to the store from my house. Both times were people pulling out of traffic, ignoring right of way. I prevented the accidents that would have resulted from these two separate idiots. I have also been in over 20 accidents in my 25 years of driving, the vast majority of those having been rear-ended and none were my fault. In my experi…

> I have also been in over 20 accidents in my 25 years of driving, the vast majority of those having been rear-ended and none were my fault. Do you just drive a lot or do you brake too late / too hard? Because an accident rate that high is rather unusual.

> or do you brake too late / too hard

You mean live in a place where drivers tailgate?

Re: Adversarial image attacks are no joke

#127

Post by p2p_astroturf »

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

Your analysis does not break out of the well known box that is the classical ways of analyzing the security of a computer system (it actually creeps into DRM/TPM territory which is known insecure despite governments with guns). Thus the security of "AI" algorithms remains as insecure as it already was, and should not be used for anything that needs to be secure. If anything, the people who make critical infrastructure insecure should go to prison (after education is reformed to actually teach these basic problems). Your example is like how typical american citizens get their panties in a bunch and throw you in jail for 5000 years if you fake your identity, but this is only because they have build such insecure systems that comeletely break down once this happened. And this is yet another thing not fixed by policing. Sorry not sorry if I sound rude. You are basically asking me to go to jail so you can use some convenient AI consumer tech in lieu of proper solutions for stuff like authentication, court systems, and car driving (and all the other thing the wackos want to replace with AI).

Re: Adversarial image attacks are no joke

#128

Post by mgraczyk »

p2p_astroturf wrote: post #127
mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

Your analysis does not break out of the well known box that is the classical ways of analyzing the security of a computer system (it actually creeps into DRM/TPM territory which is known insecure despite governments with guns). Thus the security of "AI" algorithms remains as insecure as it already was, and should not be used for anything that needs to be secure. If anything, the people who make critical infrastructur…

No, I'm asking you to go to jail if you intentionally try to cause somebody to die.

Re: Adversarial image attacks are no joke

#129

Post by thaumasiotes »

mgraczyk wrote: post #7

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Inst…

> Instead, we rely on weaker protections. We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison.

In most cases, our first and last defense against an attack of this type is to rely on the fact that nobody is interested in doing it.

Re: Adversarial image attacks are no joke

#130

Post by semi-extrinsic »

mabbo wrote: post #114
glitchc wrote: post #102

Earlier quoted context omitted.

Adversarial examples don't confuse people, only algorithms. Perhaps you need to face the fact that if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. I don't trust my life to your "It kinda looks like a road, oh wait it's a pylon, I've been tricked, BAM!" dumpster fire of an algorithm. We used to have to craft robustness i…

> if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. This is the tricky bit. Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental st…

> So how do we decide when the automated car is 'good enough'?

This is actually a really interesting point. I don't think people appreciate how far accident rates have actually dropped for modern cars without self driving. Even at million-cars-per-year sales rate you will need years of data to prove that a single self-driving software+hardware combo is better than humans with high statistical confidence. Your development cycles would be decades-long, like in aviation, if you want to be sure you're actually improving.

Post reply on HN