Another Tesla on autopilot steers towards a barrier • Hacker News Board

Re: Another Tesla on autopilot steers towards a barrier

#431

Post by gowld » Sat, Apr 07, 2018, 7:10 AM UTC

lisper wrote: post #401

thaumasiotes wrote: post #395

Earlier quoted context omitted.

> The guarantee is not that the system never enters one of those states, the guarantee is that if the system enters one of those states it never (well, so extremely rarely that it's "never" for all practical purposes) fails to recognize that it has done so. Why do you find that so implausible? Stated so, that is plausible. However, "it is possible to engineer a system that never fails to detect that it has failed" is…

> Stated so, that is plausible. However, "it is possible to engineer a system that never fails to detect that it has failed" is not They seem the same to me. In fact, in 27 years you are the first person to voice this concern. > I claim that any subset of states which is amenable to this level of detectability will exclude some other states that any normal person would also consider to be "failure". Could be, but tha…

What happens if all your sensors fail, including the one that senses the failure of your sensors? What happens if the power source disconnects?

Re: Another Tesla on autopilot steers towards a barrier

#432

Post by MertsA » Sat, Apr 07, 2018, 7:10 AM UTC

microtherion wrote: post #154

ams6110 wrote: post #12

Earlier quoted context omitted.

> Very glad the driver had 100% of their attention on the road at that moment. Yes, and 100% is critical. That required pretty quick and smooth reactions. The car started to follow the right-hand lane, then suddenly re-centered on the barrier. The driver had about a second to see that and correct. That's just the sort of situation where even a slightly inattentive driver might panic and overcorrect, which could cause…

> The driver had about a second to see that and correct. And if there had been a crash, Tesla probably would have said that the driver had an unobstructed view of the barrier for several seconds before impact (conveniently omitting at what point AP started to swerve).

And don't forget pointing out that the driver forgot to use their blinker a week before the crash so clearly they just aren't a responsible driver.

Re: Another Tesla on autopilot steers towards a barrier

#433

Post by Piskvorrr » Sat, Apr 07, 2018, 7:11 AM UTC

w_t_payne wrote: post #333

I would argue that these incidents indicate a need for aerospace levels of quality assurance. With the right tooling this does not need to be expensive. I hope that incidents like this encourage people to develop open source tooling to support safer software and higher levels of quality. We can see how lives might depend on it.

Aerospace QA is expensive by definition, because reliability nines generally are, whatever the industry: an additional nine costs 10x. You can go cheaper, but with a corresponding decrease in reliability.

Re: Another Tesla on autopilot steers towards a barrier

#434

Post by gowld » Sat, Apr 07, 2018, 7:13 AM UTC

binaryjay wrote: post #386

MattConfluence wrote: post #371

Earlier quoted context omitted.

Is it really odd that two people could reach the same conclusion when presented with the same video?

Of course not. But it is odd when the top rated comment directly below the video is essentially the armchair theory that was proposed.

Again, how is it odd that everyone agrees with the obvious explanation?

Re: Another Tesla on autopilot steers towards a barrier

#435

Post by Tomminn » Sat, Apr 07, 2018, 7:15 AM UTC

I commented on the first one that humans could easily make the mistake the Tesla made. This, however, is a scarily bad fuck up.

Re: Another Tesla on autopilot steers towards a barrier

#436

Post by aerovistae » Sat, Apr 07, 2018, 7:15 AM UTC

MadcapJake wrote: post #69

donkeyd wrote: post #32

Earlier quoted context omitted.

> I think autopilot is currently a death trap. I think this is a bit of an exaggeration. As long as the driver keeps paying attention and uses it as a driving aid, not a driver replacement, everything is fine. It's the moment that people start relying on it doing something that it wasn't built for, that the problems arise.

> I think this is a bit of an exaggeration. As long as the driver keeps paying attention and uses it as a driving aid, not a driver replacement, everything is fine. It's the moment that people start relying on it doing something that it wasn't built for, that the problems arise. I think this is a bit too optimistic. People will start relying on it to be an autopilot. I think most people see this as the desired result…

> Do you really think anyone wants "a car that will drive itself while you are paying full attention to every detail"? Otherwise, what is a driver really gaining from this? They're still expected to pay just as much attention (if not more) and I'd bet it's even more boring than regular driving (no interaction from the driver means it's like the world's most boring VR movie)

EXACTLY. They're at even more risk for accident due to inattention because it's so hard to focus on doing nothing.

Re: Another Tesla on autopilot steers towards a barrier

#437

Post by gowld » Sat, Apr 07, 2018, 7:19 AM UTC

jboles wrote: post #307

michaelt wrote: post #82

Earlier quoted context omitted.

Guess what computers excel at? Driving consistently on consistent highways. Watching the video, would you say the computer was excelling, or that the road was radically unusual? I wouldn't.

The location is the end of some tidal flow lanes rejoining the main freeway lanes with an unusual configuration of lane markings and barriers. I drive past the spot all the time and can totally see a computer messing up here.

The computer decided that staying in the middle of a wide lane was more important than avoiding a large obstacle where there was sufficient room on either side. That's bog-standard target-fixation fallacy. Is that excelling?

Re: Another Tesla on autopilot steers towards a barrier

#438

Post by Piskvorrr » Sat, Apr 07, 2018, 7:28 AM UTC

mattbeckman wrote: post #311

One way to help might be to disable AP following an update for X days or weeks. Let AP run in a disabled simulation mode while you drive your normal routes. If it determines your course of action in any situation breaks the threshold of what it was thinking it should do next, auto-report video feed to Tesla, and prevent AP from being used in a X-mile radius around that area.

It seems that with current developments, that's a complicated way of saying "turn it off."

Re: Another Tesla on autopilot steers towards a barrier

#439

Post by andosa » Sat, Apr 07, 2018, 7:50 AM UTC

lisper wrote: post #122

Here is my armchair diagnosis: right before the car veers towards the barrier it drives through a stretch of road where the only visible lane marker is on the left. Then the right lane marker comes into view at about the point where the lane starts to widen out for the lane split. The lines that will become the right and left lane markers of the split left and right lanes respectively are right next to the van in fro…

> It's not possible to engineer an autonomous system that never fails, but it is possible to engineer one in such a way that it never fails to detect that it has failed.

That sounds highly dubious. Here's a hypothetical scenario: there's a very drunk person on the sidewalk. As a human driver, you know he might act unexpectedly so you slow down and steer to the left. This will help you avoid a deadly collision as the person stumbles into the road.

Now let's take a self driving car in the same scenario, where, since it doesnt have general intelligence, it fails to distinguish the drunk person from a normal pederstrian and keeps going at the same speed and distance from the sidewalk as normally. How, in this scenario, does the vehicle 100% know that it has failed (like you say is always possible)?

Re: Another Tesla on autopilot steers towards a barrier

#440

Post by Too » Sat, Apr 07, 2018, 7:51 AM UTC

readhn wrote: post #402

Why is autopilot feature still allowed to exist? It should be disabled until further improvements. This blows my mind.

This is the biggest wtf. We've now seen at least three videos of similar trivial mistakes by autopilot only this week with one fatal crash and it is still not disabled.

When aircrafts crash all planes of that model are grounded until the root cause is found. Self driving cars need similar processes.