Live data from Hacker News

NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

arstechnica.com

481–490 of 546 posts

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#481
post #42

NTSB: • At 8 seconds prior to the crash, the Tesla was following a lead vehicle and was traveling about 65 mph. • At 7 seconds prior to the crash, the Tesla began a left steering movement while following a lead vehicle. • At 4 seconds prior to the crash, the Tesla was no longer following a lead vehicle. • At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increas…

So it should be possible to murder Tesla drivers by painting adversarial strips on the road?

Perhaps. But it's also possible to murder (or endanger) non Tesla drivers by messing with road signs as well.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#482

Earlier quoted context omitted.

All cars from all manufacturers are running 10s or 100s of times more code than they did 10 or 20 years ago. https://skeptics.stackexchange.com/questions/39559/does-the-... This really affects the nature of the situation. 20 years ago, cars contained microcontrollers with a tiny bit of code which was thoroughly reviewed and tested by skilled professionals. Today, all cars run so much code, even outside of the enterta…

When the Toyota electronic throttle "unintended acceleration" accidents were in the news, the software was described as a "big bowl of spaghetti" but the NTSB ultimately determined that it was not the cause of the problems. It was drivers using the wrong pedal.

I've long been curious about the "big bowl of spaghetti" comment (and all the other criticisms made by the experts who inspected Toyota's code). There were some extremely serious accusations which don't seem consistent with the fact that the vast majority of Toyotas on the road aren't showing problems caused by their MCU's spaghetti code.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#483
post #418

Earlier quoted context omitted.

Why are roads in the US so post-apocalyptic? Even your good example here looks horrible.

Hmm, perhaps something to do with there being more road miles in the US than in all of Europe combined?

The European road network is actually slightly larger than the US one (~6.5 million km vs ~7 million km).

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#484
post #137

Earlier quoted context omitted.

This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. So what is the right way to handle these updates? You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer. Toyota had a similar problem a few years ago and…

>This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. Cars have been dependent on software for a long time (literally decades). This isn't something new. Even combustion engine cars have had software inside of them that controls the operation of the engine, and this software is vigorously tested for safety issues (because most car manufacturers unde…

> There is no evidence that said OTA update made Tesla cars any safer.

That particular OTA update significantly shortened braking distances. [The update] cut the vehicle’s 60 mph stopping distance a whole 19 feet, to 133, about average for a luxury compact sedan. That's a safer condition, IMO, and I'm uncertain how to argue that it doesn't make the car safer.

[0] - https://www.wired.com/story/tesla-model3-braking-software-up...

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#485

Earlier quoted context omitted.

The problem, however, is your test can't be assumed to generify the way an explicitly coded web service would. You can look at your code and say "For all invalid XML, this, for all input spaces, that." You can formally prove your code in other words. You CANNOT do that with Neural Nets. Any formal proof would simply prove that your neural network simulation is still running okay. Not that it is generating the correct…

Edge cases in this scare the hell out of me. I'm envisioning watching CCTV of every Tesla that follows a specific route on a specific day merrily driving off the same cliff until it's noticed. I mean what would have happened here if another Tesla or two were directly behind Huang, following his car's lead?! Possibly nothing, I'd assume the stopping distance would be observed and the following cars would be able to st…

The report indicates the system ignores stationary objects. I would not be surprised if a suddenly decelerated car in front of the system effectively vanished from the car's situational awareness. Your scenario does not seem that far-fetched.

An attentive human would realize something horrible had happened and perhaps reacted accordingly. A disengaged or otherwise distracted one may not have the reaction time necessary to stop the system from plowing right into the situation to make it worse.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#486

Earlier quoted context omitted.

Isn’t flying mostly safe because of post hoc safety analysis followed by operating requirements? I don’t think the FAA tests every change made to aircraft before they can fly?

De facto the FAA indeed does that very thing. First, any change in design (or in configuration, in the case of repairs) is backed by PEs or A&P mechanics who sign off on the changes. Their career rides on the validity of their analysis so that's a better guarantee than some commit message by a systems programmer. Second, the FAA basically says "show us what you are changing" after which they will absolutely require p…

I think that the FAA's role is theoretically as you express, but in practice, there is significantly less oversight (especially direct oversight) than implied.

As an example, the crash of N121JM on a rejected takeoff was due (only in part) to a defective throttle quadrant/gust lock design that went undetected during design and certification, in part because it was argued to be a continuation of a conformant and previously certificated design. (Which is relevant to the current discussion in that if you decide to make certification costly and time-consuming, there will be business and engineering pressure to continue using previously certificated parts, with only "insignificant changes".)

https://aviation-safety.net/database/record.php?id=20140531-...

PS: I 100% agree on the NTSB process' contribution to safety.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#487

Earlier quoted context omitted.

And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal flying at 70mph with humans inside…

> And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal I recently got downvoted for that…

I think you got down-voted because you misrepresented what Telsa is actually doing, which is a difficult arbitrage between:

- known preventable deaths from, say, not staying in the lane aggressively enough;

- possible surprises and subsequent deaths.

There is an ethical conundrum (that is quite different from the trolley problem) between a known fix and an unknown possibility. If both are clearly establish and you dismiss the former to have a simplistic take, yes, you would be down-voted because you are making the debate less informed.

Without falling to solutionism, in that case, the remaining issue seems rather isolated to a handful of areas that look like lanes and could be either painted properly, or that Telsa cars could train to avoid. The later fix would have to be sent rapidly and could have surprising consequences -- although it seems decreasingly likely.

That learning pattern (resolving unintended surprises as they happen decreasingly often) is common in software. This explains why this community prefers (to an extend) Telsa to other manufacturers. Others have preferred the surprise-free and PR-friendly option of not saving the dozens of thousand of lives dying on the road at the moment. There are ethical backgrounds to non-interventionism.

As the victim of car violence, I happen to think that their position is unethical. I’m actually at what is considered an extreme position, of being in favour of Telsa (and Waymo) taking more risk than necessary and temporarily increasing the number of accidents on the road because they have a far better track record of learning from those accidents (and the subsequent press coverage) and that would lower the overall number of deaths faster.

As it happens, they don’t need to: even with less than half of accidents from their counterparts, they still get spectacular learning rate.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#488

Earlier quoted context omitted.

https://www.tesla.com/autopilot and best of all, guess what Tesla shows on that landing page. A driver with their hands in their lap

They do expect you to change that posture if Autopilot does something you think is dangerous, though. Wandering out of lane uninitiated (lane change indication, for example) and accelerating towards a barrier would likely count as one of those situations. I've yet to see an incident so far where a human driver, properly engaged with the driving experience and road conditions as they should be when in charge of a vehi…

> They do expect you to change that posture if Autopilot does something you think is dangerous, though.

Their lawyers and their marketing departments are contradicting each other.

Their legal department will always insist that your hands must remain on the steering wheel. If your hands are not on the wheel, and the car crashes, they will mention this clause and implicitly blame the driver.

Their marketing department (like this webpage) say "This car can drive itself!".

And the car itself? Well it has a sensor that detect hands off wheel situations (plus point from the lawyers' "cover our asses" side), but it allows a few (/lot of) seconds of that situation before it warns you (their lawyer must've once screamed "Why the fuck does the sensor tolerate this situation!").

For a stupid analogy, it's almost like a bar owner advertising his bar as a smoke joint, but saying "you're not allowed to smoke in here, and I can spot anyone smoking" but still tolerating a few minutes of lit cigarettes.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#489
I think one of my main concerns with "autopilot" is that for a lot of drivers, it will absolutely make the roads safer for them and those that use the roads around them. Consequently, for some safer and more-alert drivers, it has the potential to make driving less safe.

Re: NTSB: Autopilot steered Tesla car toward traffic barrier before deadly crash

#490

Earlier quoted context omitted.

This is a Tesla issue. Waymo uses Lidar and most likely won't run into walls.

I don't know that the issue is LIDAR, but rather distinguishing a stationary object in front from stationary objects to the side. Does Uber use LIDAR?

Well, "uses": it seems that in the case of E.H., the lidar was there but its output was discarded in the end.
Post reply on HN