NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash • Hacker News Board

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#451

Post by InternetOfStuff » Fri, Jun 08, 2018, 8:57 AM UTC

txcwpalpha wrote: post #117

falcolas wrote: post #97

Earlier quoted context omitted.

One more thing to note - anecdotal evidence indicates that Tesla cars did not attempt to center within a lane prior to an OTA update, after which multiple cars exhibited this "centering" action into gore sections (and thus required manual input to avoid an incident) on video. To me, that this behavior was added via an update makes it even harder to predict - your car can pass a particular section of road without inci…

And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal flying at 70mph with humans inside…

> And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal

I recently got downvoted for that exact line of reasoning.

Looks like some people don't like to hear that :-)

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#452

Post by mnl » Fri, Jun 08, 2018, 9:03 AM UTC

JimmyAustin wrote: post #435

TeMPOraL wrote: post #401

Earlier quoted context omitted.

> So what is the right way to handle these updates? Avoid doing them in the first place? It's not like bit rot is - or should be - a problem for cars. It's a problem specific to the Internet-connected software ecosystem, which a car shouldn't be a part of. So basically: develop software, test the shit out of it, then release. If you happen to find some critical problem later on that is fixable with software, by all m…

Tesla does test the shit out of it before they release a patch. The problem is that users expectations of the systems performance suddenly get out of sync with what the car is going to do. Part of me wonders if there should be a very quick, unskipable, animated, easy to understand explanation of the patch notes before you can drive when they make material changes to core driving functionality.

Well, then what are they testing exactly?

This is beyond broken, it's a fundamental misunderstanding of how physical products are supposed to work. Software people have gotten used to dismiss the principle of least astonishment because they know better —and no user got killed because of a Gmail redesign—, but this is a car, it's hardware with its user on-board, a lot of kinetic energy and all of it relies on muscle memory.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#453

Post by retube » Fri, Jun 08, 2018, 9:08 AM UTC

Animats wrote: post #42

NTSB: • At 8 seconds prior to the crash, the Tesla was following a lead vehicle and was traveling about 65 mph. • At 7 seconds prior to the crash, the Tesla began a left steering movement while following a lead vehicle. • At 4 seconds prior to the crash, the Tesla was no longer following a lead vehicle. • At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increas…

> These vehicles will run into stationary obstacles at full speed with no warning or emergency braking at all. That is by design

Really? Really?? I mean if was designing a self-driving system pretty much the first capability I would build is detection of stuff in the way of the vehicle. How are you to avoid things like fallen trees, or stopped vehicles, or a person, or a closed gate, or any number of other possible obstacles? And how on earth would a deficiency like that get past the regulators?

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#454

Post by agildehaus » Fri, Jun 08, 2018, 9:18 AM UTC

MBCook wrote: post #3

“[Driver’s] hands were not detected on the steering wheel for the final six seconds prior to the crash. Tesla has said that Huang received warnings to put his hands on the wheel, but according to the NTSB, these warnings came more than 15 minutes before the crash.” This kind of stuff is why I’ve lost all faith in Tesla’s public statements. What they said here was, for all intents and purposes, a flat out lie. Clearly…

Truly fixing the flaw would be to skip to Level 4 as Google decided to do, instead of shipping a system that many drivers get too comfortable with leading to accidents such as this one.

But skipping to Level 4 fairly clearly requires LIDAR at this point in history and Musk is on record saying he thinks it can be done with cameras and radar alone. So pride will prevent that.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#455

Post by YeOakAye » Fri, Jun 08, 2018, 9:27 AM UTC

salawat wrote: post #403

mortehu wrote: post #390

Earlier quoted context omitted.

I think the point is to run the two models on the same data, either simultaneously in everyone's cars, or using recorded sensor data from the past (maybe in the car while parked after a drive for privacy reasons). Initially, only the old version gets to take any action in the real world. Any difference in lane choice would then have to be justified before making the new version "live". You can do this sort of stuff w…

The problem, however, is your test can't be assumed to generify the way an explicitly coded web service would. You can look at your code and say "For all invalid XML, this, for all input spaces, that." You can formally prove your code in other words. You CANNOT do that with Neural Nets. Any formal proof would simply prove that your neural network simulation is still running okay. Not that it is generating the correct…

Edge cases in this scare the hell out of me. I'm envisioning watching CCTV of every Tesla that follows a specific route on a specific day merrily driving off the same cliff until it's noticed.

I mean what would have happened here if another Tesla or two were directly behind Huang, following his car's lead?!

Possibly nothing, I'd assume the stopping distance would be observed and the following cars would be able to stop/avoid, but I wouldn't like to bet either way. Perhaps, in some conditions, the sudden impact on the lead car would cause the second car to lose track of the rear end of the first? Would it then accelerate into it?

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#456

Post by JoshCole » Fri, Jun 08, 2018, 9:39 AM UTC

buvanshak wrote: post #437

JoshCole wrote: post #394

Earlier quoted context omitted.

There haven't even been five deaths yet in the entire history of Tesla autopilot car accidents, but there are more than three thousand deaths a day due to car accidents. Tesla's safety record actually is such that if scaled up, their would be a massive drop in the number of deaths per day. Their marketing reflects this. So does their communication on the topic. I know it can sound offensive when they say that a drive…

>Tesla's safety record actually is such that if scaled up, their would be a massive drop in the number of deaths per day... There is not enough data to do this "scaling up". So doing so would be incredibly misleading (But doesn't stop tesla's PR from doing the same). >Tesla is also a leading player in moving away from destroying our planet. The actions of this company and the persons behind this somehow does not feel…

> There is not enough data to do this "scaling up". So doing so would be incredibly misleading (But doesn't stop tesla's PR from doing the same).

There are tens of thousands of Tesla vehicles on the road, many of which have been driven for years. However, the strength of Tesla vehicles safety doesn't rest on Tesla vehicles alone. Tesla vehicles are a class of vehicle which implement driver assistance technologies. There are many other cars that do this. Independent analysis of these cars in aggregate have shown them to reduce car accident frequency and severity.

https://www.nhtsa.gov/equipment/driver-assistance-technologi...

> The actions of this company and the persons behind this somehow does not feel compatible with such a goal. I am sorry. I am just not buying it. It is more probable that this "saving the planet" narrative is something that is meant to differentiate from the competition and to attract investors.

Tesla is a leader in the renewable energy sector. There is a need for renewable energy as a consequence of climate change. Being a leading player in renewable energy means being a leading player in combating climate change. So Tesla is a leader in combating climate change. Combating climate change is an effort to save the planet. So Tesla is a leading player in the effort toward saving the planet.

At no point in the chain of logic is it necessary to call upon the motivations of Elon Musk. If someone were kill another person, the motivation for doing that deed would not change whether or not they did in fact kill someone. In the same manner, the fact that Tesla is helping to solve the problem of climate change is a fact irregardless of the motivation of its founder.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#457

Post by rad_gruchalski » Fri, Jun 08, 2018, 9:40 AM UTC

slg wrote: post #158

txcwpalpha wrote: post #156

Earlier quoted context omitted.

> That update going out immediately OTA is going to save lives compared to if Tesla waited for the cars to be serviced like other manufacturers. I don't think you can legitimately argue against that fact. There is again no evidence to support this fact. There is evidence that Tesla's OTA software updates have introduced safety issues with Tesla cars. That's a fact. Better braking distance is of course a good thing bu…

I simply don't know what to say to you if you are going to legitimately argue that shaving 20 feet off of braking distance will not make a car any safer. EDIT: The comment I was replying to was heavily edited after I responded. It originally said something along the lines of improving braking distance is good but there is no evidence that it would improve safety.

> I simply don't know what to say to you if you are going to legitimately argue that shaving 20 feet off of braking distance will not make a car any safer.

If the car can’t see the obstacle, the braking distance simply does not matter.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#458

Post by ekianjo » Fri, Jun 08, 2018, 9:45 AM UTC

amelius wrote: post #278

stickfigure wrote: post #270

Earlier quoted context omitted.

This is a popular idea: Just put someone in charge! It ignores the incentives for those gatekeepers, who are now part of the system. In practice I don't think you're going to get better updates, you're going to get "almost no updates". See also: The FDA and drug approvals.

So you want to do away with the FDA? Put BigPharma in charge and let them regulate themselves? That sounds a lot less appealing to me.

It took years for the FDA to investigate Theranos in case you are not aware. And they only did when the press started digging up. Poor, poor track record.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#459

Post by ekianjo » Fri, Jun 08, 2018, 9:47 AM UTC

TeMPOraL wrote: post #408

wooter wrote: post #386

Earlier quoted context omitted.

fdareview.org - the fda kills more people in late/non approvals than it saves through denials. this is why compassionate use has been on the rise

It does not, in any way, imply that pharmaceutical companies would do a better job if left to their own devices and market pressure .

It is not completely out of this world to imagine multiple private entities involved in pharma dossier reviews instead of having the FDA. The FDA employs tons of private consultants anyway so they bring virtually no value.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#460

Post by mikejb » Fri, Jun 08, 2018, 9:47 AM UTC

agildehaus wrote: post #454

MBCook wrote: post #3

“[Driver’s] hands were not detected on the steering wheel for the final six seconds prior to the crash. Tesla has said that Huang received warnings to put his hands on the wheel, but according to the NTSB, these warnings came more than 15 minutes before the crash.” This kind of stuff is why I’ve lost all faith in Tesla’s public statements. What they said here was, for all intents and purposes, a flat out lie. Clearly…

Truly fixing the flaw would be to skip to Level 4 as Google decided to do, instead of shipping a system that many drivers get too comfortable with leading to accidents such as this one. But skipping to Level 4 fairly clearly requires LIDAR at this point in history and Musk is on record saying he thinks it can be done with cameras and radar alone. So pride will prevent that.

It doesn't even have to be pride. If he now says they're switching to Lidars, all cars before sold with "self-driving hardware" and "full SDC is only a software and computer upgrade away" promises are screwed, which may create a huge liability for Tesla, and impact share prices significantly. As CEO, he still has to answer to shareholders and act with them in mind.