Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

arstechnica.com

Post reply on HN
Print view View original

391–400 of 546 posts

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#391

Post by acct1771 »

oh_sigh wrote: post #250
LinuxBender wrote: post #223

Disclaimer: Taboo comment ahead. Subtle bugs in self driving cars would be a simple way to assassinate people with low cost overhead. One OTA update to a target and you could probably even get video footage of the job being completed, sent to the client all in one API call. Surely by now someone must have completed a cost analysis of traditional contractors vs. having a plant at a car manufacturer. Am I the only one…

Nope(you're not the only one) - there's a pretty popular conspiracy theory that Michael Hastings(a journalist) was killed by a car hack: https://en.wikipedia.org/wiki/Michael_Hastings_(journalist)#...

Popular for good reason. Certainly read about it outside of Wikipedia

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#392

Post by kevinchen »

Tesla Autopilot should be recalled via the next OTA update.

The “Autopilot” branding implies that users need not pay attention, when in reality, the system needs interventions at infrequent but hard-to-predict times. If an engineer at Apple can’t figure it out, then the average person has no chance. Their software sets users up to fail. (Where failure means permanent disability or death.)

Inevitably, Musk fans will claim that recalling Autopilot actually makes Tesla drivers less safe. But here's the problem with Musk’s framing of Autopilot.

Sure, maybe it fails less often than humans. (We don't know whether we can trust his numbers.) But we do know that when it fails, it fails in different ways — Autopilot crashes are noteworthy because they happen in situations where human drivers would have no problem. That’s what people can’t get over. And it is why Autopilot is such a dangerous feature.

An automaker with more humility would’ve disabled this feature years ago. (Even Uber suspended testing after the Arizona crash!) With Musk, my fear is that more people will have to die before there is enough pressure from regulators / the public to pull the plug.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#393

Post by Bluestrike2 »

slg wrote: post #137
txcwpalpha wrote: post #117

Earlier quoted context omitted.

And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal flying at 70mph with humans inside…

This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. So what is the right way to handle these updates? You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer. Toyota had a similar problem a few years ago and…

While I like Tesla, I find the praise for Tesla's fast OTA update for its braking problem to be freaking terrifying.

A problem with variable stopping distances is the sort of thing that should be blindingly obvious in the telemetry data from your testing procedures. Brake systems, and ABS controls in particular, are normally rigorously tested over the course of 12-18 months in different environments and conditions.[0] That Tesla completely missed something like that suggests either their testing procedures are drastically flawed (missing something that CR was able to easily and quickly verify in different cars), that their software development process isn't meshed up with their hardware testing and validation, or a combination of the two. Neither option is a good one.

The fact that Tesla was able to shave 19 feet of their braking distances is horrifying. After months of testing different variations and changes to refine your braking systems, shaving off an extra 19 feet should be impossible. There shouldn't be any room to gain extra inches without making tradeoffs in performance in other conditions that you've already ruled out making. If there's an extra 19 feet to be found for free after a few days of dev time, you did something drastically wrong. And that's completely ignoring physical testing before pushing your new update. Code tests aren't sufficient; you're changing physical real-world behavior, and there's always a tradeoff when you're dealing with braking and traction.

Tesla is being praised by consumers and the media because, hey, who doesn't like the idea that problems can be fixed a couple days after being identified? That's great. In this case, Tesla literally made people's cars better than they were just a few days before. But it trivializes a problem with very real consequences, and I hope that trivialization doesn't extend to Tesla's engineers. Instead of talking about a brake problem, people are talking about how great the fast OTA update for the problem is. Consumers find that comforting, as the OTA updates can makes what's otherwisea pain in the ass (recalls and dealer visits for software updates) effortless.

Hell, I'm a believer in release early, release often for software. Users benefit, as do developers. At the same time, the knowledge that you can quickly fix a bug and push out an update can be a bit insidious. It's a bit of a double-edged sword in that it gives you a sense of comfort that can bite you in the ass as it trivializes the consequences of a bug. And when bug reports for your product can literally come in the form of coroner's reports, that comfort isn't a good thing for developers.

0. https://www.forbes.com/sites/samabuelsamid/2018/05/30/rapid-...

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#394

Post by JoshCole »

buvanshak wrote: post #348
megablast wrote: post #328

Earlier quoted context omitted.

It is a car company. Their products directly kill over a million people every year. The injure 10x as many. Their pollution causes a similar number of deaths. They are responsible for so many horrible side effects, including dangerous city and suburbs to people taking up more and more space. You think this one crash makes them evil?

No, it is not the crash that makes the evil. It is how they handle it. And how they put blatantly misleading marketing out there, along with poorly researched features that may or may not have lead to overconfidence from the drivers along with half baked tech, that resulted in crashes such as these.

There haven't even been five deaths yet in the entire history of Tesla autopilot car accidents, but there are more than three thousand deaths a day due to car accidents. Tesla's safety record actually is such that if scaled up, their would be a massive drop in the number of deaths per day. Their marketing reflects this. So does their communication on the topic. I know it can sound offensive when they say that a driver died without his hands on the wheel in the six seconds leading up to the crash, but their communication is reflective of a desire to preserve life. So I wouldn't call it evil, let alone call it blatantly evil.

Stepping back further, away from this accident, Tesla is also a leading player in moving away from destroying our planet. By that I mean they are pushing renewable energy. Again, not something I would call evil, let alone blatantly evil.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#396

Post by mcot2 »

Animats wrote: post #42

NTSB: • At 8 seconds prior to the crash, the Tesla was following a lead vehicle and was traveling about 65 mph. • At 7 seconds prior to the crash, the Tesla began a left steering movement while following a lead vehicle. • At 4 seconds prior to the crash, the Tesla was no longer following a lead vehicle. • At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increas…

I think we need some more details in order to conclusively blame the Autopilot for this death. I would like to see the full report and really do hope that Tesla can see their way to coming back into the fold and cooperating with the NTSB on a deeply technical fact based analysis on what happened here. For one thing, we do need to know more about the broken crash attenuator and the impact that it had on a severity of the incident.

All of that being said, I still think Tesla has mostly the right approach to their Autopilot system. There is an unacceptably high number of crashes caused by human error and getting to autonomous driving as fast as possible will save lives. It is virtually impossible to build a self driving system in a lab, with all current known methods you must have a large population of vechicles training the system. The basic calculus with their approach is that the safety risks of not getting to autonomous driving sooner is more than the risks of failures in the system along the way to getting there. It is admittedly a very fine line to walk but I do see the logic in it.

I do think that Tesla could do more to educate the users who are using early versions of the software.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#397

Post by gscott »

wpdev_63 wrote: post #395
gscott wrote: post #385

Earlier quoted context omitted.

The other poster was speaking of entrapment

Yeah but where did a thumbdrive come into play?

That would be a fictitious example of entrapment to go along with the posters idea that the person who released the nsa tools archive might have been entrapped.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#398

Post by taneq »

Pfhreak wrote: post #299
txcwpalpha wrote: post #117

Earlier quoted context omitted.

And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal flying at 70mph with humans inside…

This seems testable. IANAAE (I Am Not An Automotive Engineer), but why can't you run both the new and old code side by side and if the actions they take are materially different investigate further? Like, if in one case the new code would want you to move left, and the old code goes straight, one of those behaviors is probably wrong. If the driver corrects, then the new code is probably correct, but if the driver doe…

> why can't you run both the new and old code side by side and if the actions they take are materially different investigate further?

As I understand it, this is essentially what they were doing with the autopilot 'shadow mode' stuff. Running the system in the background and comparing its outputs with the human driver's responses, and (presumably) logging an incident when the two diverged by any significant margin?

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#399

Post by TaylorAlexander »

testplzignore wrote: post #80
mymacbook wrote: post #27

Reading that initial report is terrifying. I am so glad the NTSB set the record straight that the driver had his hands on the wheel for the majority of the final minute of travel. Really makes me feel like Tesla was out to blame the driver from the get go. To be clear the driver is absolutely partially at fault, but my goodness autopilot sped up into the barrier in the final seconds — totally unexpected when the car…

> Don’t want to see autopilot disabled as a result of this, would rather Tesla use this to double down and apply new learnings. I agree that Tesla obviously needs to be learning from this. But at this point in time, Autopilot needs to be turned off until Tesla can fix the fundamental flaws in the set of sensors on their cars. There is now strong evidence that Autopilot is dangerous, and there is still weak evidence t…

I have a hard time seeing it this way. Turned off until they can fix the fundamental flaws? Which flaws exactly? It’s a prototype technology that no one knows how to do perfectly. Should Waymo also be taken off the road? None of these technologies are provably safe, so we always have to accept some risk. How much risk is appropriate is hard to determine. Perhaps LIDAR would reduce the risk, but Uber showed you can kill people even on a LIDAR based vehicle if your software fails to classify the risk properly. And humans drive with just two eyeballs so it’s not unreasonable to think camera based systems should be in principle possible to make safe.

So we could pull all the self driving cars off the road, but then we inhibit their ability to collect the wide range of real world data necessary to build the technology. Or we could pull the less successful tech off the road sooner, but that would cause those programs to wither and hand the market to one or two well funded players.

I just don’t see how pulling the tech off the road because of a flaw that Tesla has likely already fixed would accomplish much.

I do struggle to understand what we should do, but I’m not so quick to think the tech should be pulled.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#400

Post by taneq »

abalone wrote: post #241
falcolas wrote: post #97

Earlier quoted context omitted.

One more thing to note - anecdotal evidence indicates that Tesla cars did not attempt to center within a lane prior to an OTA update, after which multiple cars exhibited this "centering" action into gore sections (and thus required manual input to avoid an incident) on video. To me, that this behavior was added via an update makes it even harder to predict - your car can pass a particular section of road without inci…

Two years ago there was an Autopark accident that made the news. Tesla blamed the driver -- very effectively.[1] But if you look closely, it likely was due to an unexpected OTA change in behavior combined with poor UI design. In the accident, the driver double-tapped the Park button which activated the Autopark feature, exited the car and walked away. The car proceeded to move forward into an object. The driver claim…

Even if there are patch notes and a clickthrough waiver, there is no possible way that they could express in words what they've changed when pushing an update to a neural net based system. Saying "you accepted the update" is ridiculous when it's not even possible to obtain informed consent.
Post reply on HN