Live data from Hacker News

NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

arstechnica.com

241–250 of 546 posts

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#241
post #42

NTSB: • At 8 seconds prior to the crash, the Tesla was following a lead vehicle and was traveling about 65 mph. • At 7 seconds prior to the crash, the Tesla began a left steering movement while following a lead vehicle. • At 4 seconds prior to the crash, the Tesla was no longer following a lead vehicle. • At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increas…

One more thing to note - anecdotal evidence indicates that Tesla cars did not attempt to center within a lane prior to an OTA update, after which multiple cars exhibited this "centering" action into gore sections (and thus required manual input to avoid an incident) on video. To me, that this behavior was added via an update makes it even harder to predict - your car can pass a particular section of road without inci…

Two years ago there was an Autopark accident that made the news. Tesla blamed the driver -- very effectively.[1] But if you look closely, it likely was due to an unexpected OTA change in behavior combined with poor UI design.

In the accident, the driver double-tapped the Park button which activated the Autopark feature, exited the car and walked away. The car proceeded to move forward into an object. The driver claimed he merely put it in park and never activated the auto-park feature. Tesla responded with logs proving he double-tapped and there were audible and visual warnings.

Well, I looked more closely. Turns out Tesla pushed an OTA update that added this "double-tap Park to Autopark" shortcut. And it's one bad design decision after another.

First, let's note the most obvious design flaw here: The difference between instructing your car to immobilize itself (park) and instructing it to move forward even if the driver was absent (Autopark) was the difference between a single and double tap on the same button. A button that you might be in the habit of tapping a couple times to make sure you've parked. So it's terrible accident-prone design from the start.

Second issue is user awareness. Normally Tesla makes you confirm terms before activating new behavior, and technically they did here, but they did it in a horribly confusing way. They buried the mere mention of the Autopark shortcut under the dialog for "Require Continuous Press".[2] So if you went to turn that off -- that's the setting for requiring you to hold down a FOB button during Summon -- and you didn't read that dialog closely, you would not know that you'd also get this handy-dandy double-tap Autopark shortcut.

Third is those warnings. Useless. They did not require any kind of confirmation. So if you "hit park" and then quickly exited your vehicle, you might never hear the warning or see it on the screen that, by the way, that shortcut that you didn't know existed just got triggered and is about to make the car start moving forward while you walk away.

So I think it's quite plausible that the driver was not at fault here -- or at least that it was a reasonable mistake facilitated by poor design. It's unfortunate that Tesla was able to convince so many with a "data dump" that the driver was clearly at fault.

I still recall that poor NYT journalist that Tesla "caught driving in circles"[3] -- while looking for a hard-to-find charger at night. Now I hope we are developing a healthier skepticism to this (now standard) response from Tesla and look more deeply at potential root causes.

[1] https://www.theverge.com/2016/5/11/11658226/tesla-model-s-su...

[2] https://youtu.be/Cg7V0gnW1Us?t=2m10s

[3] https://www.forbes.com/sites/joannmuller/2013/02/14/teslas-g...

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#242

Earlier quoted context omitted.

>This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. Cars have been dependent on software for a long time (literally decades). This isn't something new. Even combustion engine cars have had software inside of them that controls the operation of the engine, and this software is vigorously tested for safety issues (because most car manufacturers unde…

>Cars have been dependent on software for a long time (literally decades). This isn't something new. Even combustion engine cars have had software inside of them that controls the operation of the engine, and this software is vigorously tested for safety issues (because most car manufacturers understand a fault with such software could result in someone's death). Tesla seems to be the only major car manufacturer that…

> The TACC offered by most (if not all) manufacturers can't differentiate between the surroundings and stopped vehicles.

Software should not be driving a car into any of them. I think that LIDAR would see the obstacle, but as I understand, the crashed Tesla car didn't have it.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#243
post #3

“[Driver’s] hands were not detected on the steering wheel for the final six seconds prior to the crash. Tesla has said that Huang received warnings to put his hands on the wheel, but according to the NTSB, these warnings came more than 15 minutes before the crash.” This kind of stuff is why I’ve lost all faith in Tesla’s public statements. What they said here was, for all intents and purposes, a flat out lie. Clearly…

https://www.tesla.com/autopilot

and best of all, guess what Tesla shows on that landing page. A driver with their hands in their lap

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#244
post #175

Earlier quoted context omitted.

Makes me wonder about the future of self-driving cars. I would think that the first thing that should be programmed in is to not run into objects that are not moving. IMO, maybe the roads need to be certified for self-driving. Dangerous areas would be properly painted to prevent recognition errors. Every self-driving system would need to query some database to see if a road is certified. If not, the self-driving syst…

Tesla claims their cars have all the hardware required for full self-driving, but their cars do not have LIDAR. Every serious self-driving outfit understands LIDAR to be necessary. Tesla's marketing department would have people believe otherwise of course.

Exactly. I think a LIDAR could see the obstacle in this case.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#245
> During the 18-minute 55-second segment, the vehicle provided two visual alerts and one auditory alert for the driver to place his hands on the steering wheel. These alerts were made more than 15 minutes prior to the crash.

If your hand is always supposed to be on the wheel, why is does the car not constantly alert you when it detects that your hands are off (similar to how cars beep at you if your seatbelt is unbuckled while driving)?

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#246
post #110

Earlier quoted context omitted.

We don't have exact numbers for Tesla but there are 2 confirmed casualties in what is conservatively over a billion Autopilot miles. The average casualty rate for human drivers is something around 1.25 death per 100 million miles. Those numbers are not directly comparable given that Autopilot is not something that is used during 100% of all driving miles and Tesla drivers are not average car drivers. However, there i…

Can you point me to a source for the billion Autopilot miles? This is something I've been trying to track down without any luck so far. The best source I've been able to find so far is a tweet by Elon Musk in 2016 saying "Cumulative Tesla Autopilot miles now at 222 million". [1] https://twitter.com/elonmusk/status/784487348562198529?ref_s...

I wish Tesla would update those numbers more frequently, but it looks like they stopped doing that in 2016. However they did enough updates in 2016 that allow us to estimate a trend of something on the order of 1 million miles per day.

4/9/2016 - 47 [1]

5/24/2016 - 100 [2]

6/30/2016 - 130 [3]

10/7/2016 - 222 [4]

11/13/2016 - 300 [5]

This rate continuing linearly would put us at roughly 900 million today. A linear accumulation of miles ignores that Tesla has roughly twice the cars on the road that they had in 2016 when those numbers were published. I have no idea their current mileage total has reached, but I feel safe in saying it is over a billion miles.

[1] - https://twitter.com/Tesla/status/718845153318834176

[2] - https://www.theverge.com/2016/5/24/11761098/tesla-autopilot-...

[3] - https://www.tesla.com/blog/tragic-loss

[4] - https://twitter.com/elonmusk/status/784487348562198529

[5] - https://electrek.co/2016/11/13/tesla-autopilot-billion-miles...

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#247
post #73

Earlier quoted context omitted.

Yes, but the radar has poor angular resolution (but a good idea of relative velocity), so it cannot tell the difference between a stationary object at the side of the road and one in the middle of it, so it must ignore all stationary object (by ignoring all objects with an apparent velocity approximately equal to the speed of the vehicle) in order to avoid constant false positives.

Can you explain further what "poor angular resolution" means?

It cannot see small details. So it probably "sees" another car as a blurred spot and only when it is close.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#248
post #42

NTSB: • At 8 seconds prior to the crash, the Tesla was following a lead vehicle and was traveling about 65 mph. • At 7 seconds prior to the crash, the Tesla began a left steering movement while following a lead vehicle. • At 4 seconds prior to the crash, the Tesla was no longer following a lead vehicle. • At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increas…

Although I don't work in self-driving cars, I do know a fair amount of ML and AI, and I have to be honest: if my bosses asked me to build this system, I would have immediately pointed out several problems and said that this is not a shippable product.

I expect any system that lets me drive with my hands off the wheels for periods of time deals with stationary obstacles.

What is being described here, if it's correct, is a literal "WTF" compared to how Autopilot was pitched.

I wouldn't be surprised if the US ultimately files charges against Tesla for wrongful death by faulty design and advertising.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#249
post #79

> During the 18-minute 55-second segment, the vehicle provided two visual alerts and one auditory alert for the driver to place his hands on the steering wheel. These alerts were made more than 15 minutes prior to the crash. Whoah. So there were NO alerts for 15 minutes prior to the crash. Compare this with Tesla's earlier statement: > The driver had received several visual and one audible hands-on warning earlier in…

It's a blatant lie because they knew exactly what they were implying and it wasn't true, i.e, it was a lie.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#250

Disclaimer: Taboo comment ahead. Subtle bugs in self driving cars would be a simple way to assassinate people with low cost overhead. One OTA update to a target and you could probably even get video footage of the job being completed, sent to the client all in one API call. Surely by now someone must have completed a cost analysis of traditional contractors vs. having a plant at a car manufacturer. Am I the only one…

Nope(you're not the only one) - there's a pretty popular conspiracy theory that Michael Hastings(a journalist) was killed by a car hack:

https://en.wikipedia.org/wiki/Michael_Hastings_(journalist)#...

Post reply on HN