Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waymo recalls software after two self-driving cars hit the same truck

cnn.com

Post reply on HN
Print view View original

31–40 of 82 posts

Re: Waymo recalls software after two self-driving cars hit the same truck

#31

Post by Ukv »

brk wrote: post #27
im3w1l wrote: post #4

This is my biggest fear with self-driving cars. Correlated failures. As a society we are extremely good at dealing with independent accidents. We can calculate very precisely how many people will die of traffic in a given year and we can account for it, we can have insurances, and we can decide exactly how much we are willing to spend to save a life on the margin. But if everything is fine, everything is fine, everyt…

My fear is similar, but more along the lines of adversarial attacks as various weaknesses are exposed. Imagine people taking advantage of zero-day exploits that cause self driving cars to veer off the road, stop suddenly, collide, etc. It is really not that far fetched. This technology is very far away from maturity, IMO.

Relevant XKCD: https://xkcd.com/1958/

Re: Waymo recalls software after two self-driving cars hit the same truck

#32

Post by cameldrv »

carbonatedmilk wrote: post #16

One of the best things I've learnt recently is how to apply the zero blame, process improvement approach that (many) air safety regulators take to my own teams. I'd sat through 'five whys' style postmortems before, but it was reading air safety investigation reports that finally got me to understand it and make it a useful part of how we get better at our jobs. By comparison, the way we're investigating and respondin…

My guess is that Waymo does do this internally.

One difference with this situation in terms of the public perception/discussion though is that, say in the 1960s, air safety wasn't very good compared to today, but still there was no question of eliminating air travel altogether due to safety issues. Today there is definitely an anti-self-driving contingent that would like to hype up every accident to get the self driving companies shut down entirely.

Re: Waymo recalls software after two self-driving cars hit the same truck

#33

Post by bobsomers »

pests wrote: post #15
Mageek wrote: post #6

Earlier quoted context omitted.

The article says it: “We determined that due to the persistent orientation mismatch of the towed pickup truck and tow truck combination, the Waymo AV incorrectly predicted the future motion of the towed vehicle.” It was detected, but it predicted the truck would move in a way that it didn’t end up moving.

How do you drive into a solid war if you have lidar though? To say nothing of predictions, the object is where it's at.... where it's at at that moment. You don't need to predict where it's at now... because you know where it's at.

You can't drive if you only use the current "frame" of data as the basis for your decision. Imagine driving on the highway, a comfortable distance behind a lead vehicle.

The planning software would want to slam on the brakes without predicting that the blob of sensor data in front of you is going to continue moving forward at highway speeds. That motion prediction enables the planning software to know that the space in front of your vehicle will be unoccupied by the time you reach it.

A similar prediction error was the reason Cruise rear ended the bendy bus in SF a while back. It segmented the front and rear halves of the bus as two separate entities rather than a connected one, and mispredicted the motion of the rear half of the bus.

Re: Waymo recalls software after two self-driving cars hit the same truck

#34

Post by lazide »

pests wrote: post #15
Mageek wrote: post #6

Earlier quoted context omitted.

The article says it: “We determined that due to the persistent orientation mismatch of the towed pickup truck and tow truck combination, the Waymo AV incorrectly predicted the future motion of the towed vehicle.” It was detected, but it predicted the truck would move in a way that it didn’t end up moving.

How do you drive into a solid war if you have lidar though? To say nothing of predictions, the object is where it's at.... where it's at at that moment. You don't need to predict where it's at now... because you know where it's at.

You can’t drive more than a few MPH unless you’re reacting based on the expected future, rather than the current one.

It’s why it’s so difficult to do (actually) and the ability to do it well is just as much about the risk appetite of the one responsible as anything else - because knowing if a car is likely to pull out at the light into traffic, or how likely someone is to be hiding in a bush or not is really hard. But that is what humans deal with all the time while driving.

Because no one can actually know the future, and predicting the future is fundamentally risky. And knowing when to hold ‘em, and when to fold ‘em is really more of an AGI type thing.

Re: Waymo recalls software after two self-driving cars hit the same truck

#35

Post by nicbou »

im3w1l wrote: post #4

This is my biggest fear with self-driving cars. Correlated failures. As a society we are extremely good at dealing with independent accidents. We can calculate very precisely how many people will die of traffic in a given year and we can account for it, we can have insurances, and we can decide exactly how much we are willing to spend to save a life on the margin. But if everything is fine, everything is fine, everyt…

On the other hand, the cars were recalled, a team will study what happened, the problem will get fixed, simulations and tests will be created.

When I make a dumb mistake, the other drivers rarely learn from it.

Re: Waymo recalls software after two self-driving cars hit the same truck

#36

Post by nytesky »

Ukv wrote: post #31
brk wrote: post #27

Earlier quoted context omitted.

My fear is similar, but more along the lines of adversarial attacks as various weaknesses are exposed. Imagine people taking advantage of zero-day exploits that cause self driving cars to veer off the road, stop suddenly, collide, etc. It is really not that far fetched. This technology is very far away from maturity, IMO.

Relevant XKCD: https://xkcd.com/1958/

Your run of the mill hacker sure, but adversarial nations about to invade another nation and want to keep the Western World occupied? DC was brought to a crisis by the sniper; can you imagine if cars suddenly started plowing through school pickup lines or concert venues with no way for occupant to stop it? Or even without an occupant?

Re: Waymo recalls software after two self-driving cars hit the same truck

#37

Post by sokoloff »

pests wrote: post #15
Mageek wrote: post #6

Earlier quoted context omitted.

The article says it: “We determined that due to the persistent orientation mismatch of the towed pickup truck and tow truck combination, the Waymo AV incorrectly predicted the future motion of the towed vehicle.” It was detected, but it predicted the truck would move in a way that it didn’t end up moving.

How do you drive into a solid war if you have lidar though? To say nothing of predictions, the object is where it's at.... where it's at at that moment. You don't need to predict where it's at now... because you know where it's at.

In self-driving, you are making predictions about where the object is right now based on the synthesis of data from your sensors (and often filtering information from past estimates of the object position). These might be high-precision, high-accuracy predictions, but they're predictions nonetheless.

(It's been quite some years since I worked on vision-based self-driving, so my experience is non-zero but also quite dated.)

Re: Waymo recalls software after two self-driving cars hit the same truck

#38

Post by sokoloff »

esafak wrote: post #18
carbonatedmilk wrote: post #16

One of the best things I've learnt recently is how to apply the zero blame, process improvement approach that (many) air safety regulators take to my own teams. I'd sat through 'five whys' style postmortems before, but it was reading air safety investigation reports that finally got me to understand it and make it a useful part of how we get better at our jobs. By comparison, the way we're investigating and respondin…

Where did you learn about that approach, and how is it different?

Not GP, but:

There is an explicit culture and mechanism of blamelessness around safety concerns and minor violations/deviations, which is incredibly helpful. Read about the ASRS* program (admin'd by NASA, with anonymity for non-intentional issues, prohibition on use of submissions for enforcement purposes, and explicit "get out punishment" card from the FAA): https://asrs.arc.nasa.gov/overview/immunity.html

The FAA is also explicitly including "evidence of voluntary compliance" and "just culture" in its approach to aviation safety, and explicitly changed its goal from enforcement and proof to ensuring compliance [with enforcement being only one available tool]: https://www.faa.gov/about/initiatives/cp (PDF pres: https://www.faa.gov/sites/faa.gov/files/2022-09/The%20Compli... )

I'd also read a bunch of aviation reports: https://www.ntsb.gov/Pages/monthly.aspx (more detailed reports are available approximately 2 years after the occurrence date and more details are available for fatal or air carrier occurrences, so if you don't care which ones to read, filter for those to start).

If you're more video oriented, watch @blancolirio, @NTSBgov, @AirSafetyInstitute, or @pilot-debrief. (I'd skip @ProbableCause-DanGryder.)

For a short summary, there is an intense focus on determining the facts (who, what, when, where, maybe some guesses as to why) and drawing conclusions about primary and contributing causes from there.

* Aviation Safety Reporting System

Re: Waymo recalls software after two self-driving cars hit the same truck

#40

Post by awhitby »

Out_of_Characte wrote: post #30
avalys wrote: post #17

Earlier quoted context omitted.

It wasn’t a solid wall, it was another vehicle. The brakes don’t respond immediately - you need to be able to detect that a collision is imminent several seconds before it actually occurs. This means you have to also successfully exclude all the scenarios where you are very close to another car, but a collision is not imminent because the car will be out of the way by the time you get there. Yes, at some point before…

My brakes respond immediatly,for all intents and purposes as a human with a reaction time. I'm at fault if the person in front of me stops and I dont have the stopping distance to avoid a collision. I get that self driving software is difficult. But theres no excuse for this type of accident.

That might be true for the simple case of following within a lane, although you only have to drive around to realize most drivers do not leave adequate following distance at all times to make this a pure physics problem. And neither is a good driver watching only the car in front, but also the brake lights of the cars in front of that, to help anticipate the car in front's likely actions.

But take an even slightly more complex example: you're on a two lane roadway and the car in the other lane changes into your lane, leaving inadequate stopping distance for you. You brake as hard as you safely can (maybe you have a too-close follower, too), but still there will be a few seconds when you could not, in fact, avert a collision if for some reason the car in front braked.

I have no idea what the legal situation would be: is it their fault if the crash happens within 3 seconds but yours if it happens after you've had time but failed to restablish your needed stopping distance?

Honestly even in the simple one lane case, I doubt you can slam your brakes on the interstate for no reason then expect to avoid any liability for the crash, blaming your follower for following too close.

Driving has a bunch of rules, then an awful lot of common sense and social interaction on top of them to make things actually work.

Post reply on HN