NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash • Hacker News Board

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#261

Post by freerobby » Thu, Jun 07, 2018, 11:10 PM UTC

brigade wrote: post #147

rhacker wrote: post #49

Earlier quoted context omitted.

I get that in 5-10 years we MAY have SDVs that are superior than human drivers. But I think a lot of people have a mental model of SDVs _right now_ being superior to human drivers. I can't remember the crash stats, but I think the only way to know for sure is to calculate the number of miles a single human generally drives on average before they are involved in a collision or crash, etc... Now I don't have the number…

Tesla might not be advertising L5, but they're certainly claiming that Autopilot is safer than human drivers right now.

They're claiming that human+Autosteer+TACC is safer than a human. They're not claiming that Autopilot while you use your phone or watch a movie is safer than human driving. And they remind you of this every time you engage the feature.

Consider AEB. AEB is safety improvement on pure human driving. But it'd be hugely foolish to rely 100% on AEB to do your braking for you. It's not designed for that -- it can help you in a lot of situations, but not all of them. The same is true for Autopilot.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#262

Post by ataturk » Thu, Jun 07, 2018, 11:12 PM UTC

Animats wrote: post #42

NTSB: • At 8 seconds prior to the crash, the Tesla was following a lead vehicle and was traveling about 65 mph. • At 7 seconds prior to the crash, the Tesla began a left steering movement while following a lead vehicle. • At 4 seconds prior to the crash, the Tesla was no longer following a lead vehicle. • At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increas…

No matter what, the marketing use of the word "autopilot" is complete negligence. Why Tesla isn't being sued out of existence is beyond my reckoning. The liability here is enormous.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#263

Post by ataturk » Thu, Jun 07, 2018, 11:14 PM UTC

txcwpalpha wrote: post #146

slg wrote: post #137

Earlier quoted context omitted.

This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. So what is the right way to handle these updates? You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer. Toyota had a similar problem a few years ago and…

>This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. Cars have been dependent on software for a long time (literally decades). This isn't something new. Even combustion engine cars have had software inside of them that controls the operation of the engine, and this software is vigorously tested for safety issues (because most car manufacturers unde…

The software you speak of is incredibly simplistic, though. Engine management is literally a table of data that matches inputs with what to do. It is not sophisticated software.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#264

Post by qaq » Thu, Jun 07, 2018, 11:14 PM UTC

amelius wrote: post #251

slg wrote: post #137

Earlier quoted context omitted.

This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. So what is the right way to handle these updates? You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer. Toyota had a similar problem a few years ago and…

> So what is the right way to handle these updates? Require updates to be sent to a government entity, which will test the code for X miles of real traffic, and then releases the updates to the cars. Of course, costs of this are to be paid by the company.

Seriously ? so instead of taking x amount of time it will take x amount of time + a few years.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#265

Post by CamperBob2 » Thu, Jun 07, 2018, 11:16 PM UTC

amelius wrote: post #251

slg wrote: post #137

Earlier quoted context omitted.

This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. So what is the right way to handle these updates? You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer. Toyota had a similar problem a few years ago and…

> So what is the right way to handle these updates? Require updates to be sent to a government entity, which will test the code for X miles of real traffic, and then releases the updates to the cars. Of course, costs of this are to be paid by the company.

That sounds like a great way to stall all further progress, which has a horrific human cost of its own.

Government has a valid role to play, though, by requiring full disclosure of the contents of updates and "improvements," by setting and enforcing minimum requirements for various levels of vehicle autonomy, and by mandating and enforcing uniform highway marking standards. Local DOTs are a big part of the problem.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#266

Post by swiley » Thu, Jun 07, 2018, 11:16 PM UTC

Animats wrote: post #42

NTSB: • At 8 seconds prior to the crash, the Tesla was following a lead vehicle and was traveling about 65 mph. • At 7 seconds prior to the crash, the Tesla began a left steering movement while following a lead vehicle. • At 4 seconds prior to the crash, the Tesla was no longer following a lead vehicle. • At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increas…

“Car firmware shouldn’t be open source, that’s dangerous.”

There’s no way in hell Tesla would have gotten away with selling this for so long if their users were allowed to read the unobfuscated source.

The fact that people are given no control over these things that can kill them while the manufacturers can just mess around with it without any real oversight is absolutely insane. I really don’t think the average IRC lurker who could figure out how to compile the firmware could be any more dangerous than the “engineers” who wrote it in the first place.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#267

Post by freerobby » Thu, Jun 07, 2018, 11:19 PM UTC

ericpauley wrote: post #211

falcolas wrote: post #112

Earlier quoted context omitted.

> To be clear the driver is absolutely partially at fault I'm... not so certain. Why? The autopilot had likely exhibited proper behavior every time that the vehicle had passed that particular section of road prior, and if the driver was paying full attention to the behavior of the vehicle, he would only notice the problem around the 5 second mark. Five seconds, if you have no reason to be concerned about the vehicle'…

My vehicle (a Honda civic) exhibits this exact same functionality and behavior. (lane keeping, ACC, emergency braking for cars) They make it very clear the limitations of these systems. I'd say that in the 5 months I've owned it it's had this exact behavior (veering off exit ramps) 10 times. A simple jerk of the wheel puts it back on track, it's such a natural motion if you're paying even the slightest bit of attenti…

Tesla reminds you to pay attention and keep your hands on the wheel every time you engage Autopilot. It's one of very few legal disclaimers they show you all the time on the screen, and don't give you any way of turning it off.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#268

Post by jonknee » Thu, Jun 07, 2018, 11:20 PM UTC

ingenieroariel wrote: post #24

devy wrote: post #7

Earlier quoted context omitted.

Add to that, the last bullet point from page 2 of the official preliminary report[1]: At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increased from 62 to 70.8 mph, with no precrash braking or evasive steering movement detected. That to me is a strong indicator that Tesla's AP2 can not recognize a crashed attenuator, probably one of the strongest argument that…

This happens with regular adaptive cruise control on my Ford, when the cruise control set point is a lot higher than the current speed: the car accelerates after another car clears the lane and ram into things.

Like all adaptive cruise controls, it will accelerate if you have the max speed set higher, but it will not ram into things since it's on you to steer.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#269

Post by freerobby » Thu, Jun 07, 2018, 11:21 PM UTC

falcolas wrote: post #112

mymacbook wrote: post #27

Reading that initial report is terrifying. I am so glad the NTSB set the record straight that the driver had his hands on the wheel for the majority of the final minute of travel. Really makes me feel like Tesla was out to blame the driver from the get go. To be clear the driver is absolutely partially at fault, but my goodness autopilot sped up into the barrier in the final seconds — totally unexpected when the car…

> To be clear the driver is absolutely partially at fault I'm... not so certain. Why? The autopilot had likely exhibited proper behavior every time that the vehicle had passed that particular section of road prior, and if the driver was paying full attention to the behavior of the vehicle, he would only notice the problem around the 5 second mark. Five seconds, if you have no reason to be concerned about the vehicle'…

> I'm... not so certain. Why?

In this specific case, because if a driver is watching the road in front of them, four seconds is an awful lot of time to watch yourself head toward and then accelerate into a cement barrier, all without touching the brakes. It's fine to want Autopilot to be better, but as a matter of law, it is the driver's responsibility to slam on the brakes in that situation, and move into a legal lane.

More broadly, it's because the contract of an L2 system is that a human is in control at all times. L2 systems are assistive and not autonomous. They will never disobey a driver's presets, nor override a driver's real-time inputs, even if the car thinks it is safe(r) to do so. This is a major design principle behind every L2 system, and the reason why there are no scenarios where an L2 system is considered at fault by law.

Now obviously, if there were a bug that caused an L2 system to override a user's input -- say disregarding someone hitting the brakes, or overpowering the steering wheel, that'd be a systemic failure and grounds for a recall. But we haven't seen any cases of that, and there are hardware precautions (e.g. limited torque in the Autosteer servo) to minimize that possibility.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#270

Post by stickfigure » Thu, Jun 07, 2018, 11:27 PM UTC

amelius wrote: post #260

ekianjo wrote: post #256

Earlier quoted context omitted.

Yeah, because we know governments are really good at giving certifications and doing tests that mean sonething. Lets put every design decision in the hand of governements then! or better, nationalize car companies! Problem solved?

In any case, let a third party control the actual updating, so that we know when and how often cars are updated. Require at least X months of time between code submission and deployment to cars. We don't want a culture of "quick fixes".

This is a popular idea: Just put someone in charge! It ignores the incentives for those gatekeepers, who are now part of the system. In practice I don't think you're going to get better updates, you're going to get "almost no updates".