Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why collision avoidance is harder for an AI-based system

medium.com

Post reply on HN
Print view View original

61–70 of 145 posts

Re: Why collision avoidance is harder for an AI-based system

#61

Post by jmalicki »

jdmichal wrote: post #2

> AI is not preprogrammed to monitor a known input from a sensor to take a predefined action. I guess one of my outstanding questions, which reading this only confirmed, is why this is the case? I mean, humans are pretty good examples of intelligence. And yet we still have and use these anti-collision systems. Because, in the end, when wrong decisions are made these systems save lives. Why would AI-driven vehicles no…

" I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself?"

I think that's a reasonable definition of ABS - an AI (in most cases for ABS, an expert system AI) that knows how to best mediate the brakes, with levels of performance that exceed the vast majority of humans.

Re: Why collision avoidance is harder for an AI-based system

#62

Post by scrumper »

phyzome wrote: post #41
decacorn wrote: post #6

Earlier quoted context omitted.

in response to that piece by Brad, I sincerely hope that the "safety" driver in the uber accident was fired.

The safety driver is basically irrelevant, beyond being a scapegoat. I've written up some words on this: https://www.brainonfire.net/blog/2018/03/23/safety-driver-sh...

I mAde a similar comment on the video link last week. The good news is that the NTSB is extremely good at human factors stuff - google Cockpit Resource Management for a fascinating trip down the aviation safety rabbit hole.

I sadly agree with you on the scapegoat point.

Re: Why collision avoidance is harder for an AI-based system

#63

Post by seanmcdirmid »

hangonhn wrote: post #59
seanmcdirmid wrote: post #51

Earlier quoted context omitted.

Integrating AI with a subsumption architecture sounds interesting but could still not work. You would be abstracting a lot of details (as well as potential action) from the training process, so the overall result you get still might not work. Training on everything might actually yield a better result. Does anyone know how iRobot (Brooks' company) does it?

I believe it is subsumption based with random walk added on top: bumper hit means turn around and wheel dangling means don't go farther. I think the more recent models have more intelligence added on top but still has the lower level systems acting as safe guards. Of course it doesn't run into the problems you mentioned because it doesn't have any learning (at least not one of the earlier models I had).

iRobot builds more than just roomba, I was wondering about their more advanced military robots.

Re: Why collision avoidance is harder for an AI-based system

#64

Post by jdmichal »

CardenB wrote: post #39
jdmichal wrote: post #2

> AI is not preprogrammed to monitor a known input from a sensor to take a predefined action. I guess one of my outstanding questions, which reading this only confirmed, is why this is the case? I mean, humans are pretty good examples of intelligence. And yet we still have and use these anti-collision systems. Because, in the end, when wrong decisions are made these systems save lives. Why would AI-driven vehicles no…

The false positive rate of this system could be very noisy. At what point should you brake, even if the AI says not to? When you might collide with a lidar point? You want to collide with lidar points all the time (the ground beneath the vehicle, for example, or any reflective noise). The AI is there to tell you what lidar points are actual obstacles that you can't collide with, otherwise it's not really meaningful.

This line of argument doesn't mean anything when we already have cars with intelligent, independent drivers, and these systems. And yes, their purpose would be to override the AI control when they trigger. Exactly the same as they do right now.

Re: Why collision avoidance is harder for an AI-based system

#65

Post by Animats »

This is looking at it all wrong. Watch Chris Urmson's video from SXSW on how Waymo does it. I've mentioned that before. You build a map of what's around the vehicle. Then you try to tag some obstacles to help predict their behavior. But the obstacle detection is geometric, and does not depend on the tagging, which uses machine learning. If there's an obstacle, the system doesn't hit it, even if it has no clue how to identify it.

Tesla/Mobileye managed to get that backwards. Their original system was "recognize vehicle visually, compute distance and closing rate to vehicle". If it didn't recognize an obstacle as a vehicle, it ignored it. We know this for sure, because you can buy a Mobileye unit as a dashcam-like warning device and many people have seen how they work. That led to three collisions with vehicles partly blocking the left side of a lane. One death ramming a street sweeper, one collision with a stopped fire truck, one sideswipe. The NTSB is investigating the fire truck collision.

The NTSB is now investigating the Uber collision.[1] As they usually do, the first thing they did was to get control of the wreckage.[2] Uber does not have control of the investigation. The NTSB investigators are working this like an air crash. They are "beginning collection of any and all electronic data stored on the test vehicle or transmitted to Uber". As usual, they haven't announced much, but they have mentioned that the video seen publicly is from a third-party dashcam, not the vehicle sensors.

[1] https://www.ntsb.gov/news/press-releases/Pages/NR20180320.as...

[2] http://wsau.com/news/articles/2018/mar/21/arizona-police-rel...

Re: Why collision avoidance is harder for an AI-based system

#66

Post by Retric »

seanmcdirmid wrote: post #51
hangonhn wrote: post #44

Hang on. That argument makes no sense. We've had subsumption architecture for a long time now( https://en.wikipedia.org/wiki/Subsumption_architecture ). Subsumption architecture puts some of the intelligence in the lower level systems. While the higher level controls can tell the lower level systems what it wants, it can't do things that the lower system determines is dangerous. So if a normal Mercedez has a collisio…

Integrating AI with a subsumption architecture sounds interesting but could still not work. You would be abstracting a lot of details (as well as potential action) from the training process, so the overall result you get still might not work. Training on everything might actually yield a better result. Does anyone know how iRobot (Brooks' company) does it?

The AI can still drive the car normally, the only thing that changes is it's prevented from running into another car at full speed. Which is no worse than having drivers override the system forcing a disengagement.

Re: Why collision avoidance is harder for an AI-based system

#67

Post by ltrcola »

icc97 wrote: post #17

> Also, lets back up our AI’s with old school collision avoidance! Intelligence is not the same as perfection, at least for now. The car was travelling at 38mph and never braked. Even if the collision-avoidance only saw it at the last moment it would still have braked and potentially slowed the car enough so that the woman was injured instead of being killed. I'm all for self-driving and fully believe it can improve…

Humans can't properly detect the most vulnerable users in all conditions. Not an excuse for this system failing to perform, but all self-driving cars have to do is be a little bit better for net safety to improve. People are notoriously unreliable. Computers don't get drunk or sleepy, for instance.

Re: Why collision avoidance is harder for an AI-based system

#68

Post by Animats »

jvanderbot wrote: post #49
hangonhn wrote: post #44

Hang on. That argument makes no sense. We've had subsumption architecture for a long time now( https://en.wikipedia.org/wiki/Subsumption_architecture ). Subsumption architecture puts some of the intelligence in the lower level systems. While the higher level controls can tell the lower level systems what it wants, it can't do things that the lower system determines is dangerous. So if a normal Mercedez has a collisio…

Yep! Every single autonomous system I'm aware of (and that I'm allowed to speak about) has such a system designed in from the ground up. The article presupposes we allow "intelligence" handle all decision making, which seems ridiculous even in biological systems, let alone those we design for a single purpose.

Yes. We had that in our DARPA Grand Challenge vehicle a decade ago. In addition to the main navigation and control system, there was a separate process taking inputs from the radar and speedometer, with the authority to slam on the brakes if the vehicle was about to hit something. Backing that up was a hardware stall timer reset every 100ms. If it hit 120ms without being toggled by the computer watchdog process, the brakes went on and the engine dropped to idle with no computer involvement. Then there was a DARPA-mandated remote engine kill radio system.

Re: Why collision avoidance is harder for an AI-based system

#69

Post by Animats »

SlowRobotAhead wrote: post #5

Has anyone been able to even remotely explain why the LIDAR system wasn't going nuts? I saw the "it was dark" nonsense, but I assume this vehicle had laser and IR right? The camera footage was released, I'd like to see the lidar representation.

The camera video seen publicly is from a dashcam, not the vehicle's own sensors. The NTSB has mentioned this.

Re: Why collision avoidance is harder for an AI-based system

#70

Post by ltrcola »

arithma wrote: post #37

Why isn't car driving AI put to the same rigour of testing and approval as that of airplanes? There are far more people on the road than in the sky, and yet, there's formal verification for airplane autopilot code but an accident like this is supposed to be "you couldn't have prevented it either?" If no one could prevent such a thing, this AI should never drive a car!

More rigor is definitely required, but it may not be possible to formally verify AI driven code to the same extent. Aviation software is driven from formal requirements, has a strict set of coding rules, and the generated assembly code is compared to the original source code. Within the coding rules, the software isn't even allowed to dynamically allocate memory.

I have no idea how you'd build an AI system with those constraints, given that the computer essentially programmed the model itself by learning.

Post reply on HN