After 6 months of working fine, Tesla software update drives at barriers again • Hacker News Board

Re: After 6 months of working fine, Tesla software update drives at barriers again

#331

Post by lisper » Thu, Mar 21, 2019, 4:42 PM UTC

Domenic_S wrote: post #328

lisper wrote: post #128

Earlier quoted context omitted.

> Your hands. Sure, if you react fast enough. Let's not lose the plot here. The original claim was: "The AP can only kill you if you're distracted/asleep at the wheel and not paying attention." (Emphasis added.) And that's not true. It can kill you, quite simply, by producing the wrong control input in a situation where the available recovery time is less than your reaction time. If you doubt this, then I challenge y…

This seems like a meaningless argument - we use fly-by-wire systems all the time and your point is true for most of them. Should we be suspicious of electronic throttle because it could theoretically hit the gas at a crosswalk when you tried to stop?

Should we be suspicious of any automation that can cause death and injury if it doesn't work right? Yes. Absolutely. These systems need to be very carefully vetted. When you don't do that, you end up with cars driving into cyclists [1] and barriers [2], and airplanes driving themselves into the ground [3]. Automation that puts civilian lives in danger is not something where you can "move fast and break things" because the things you end up breaking are innocent human beings.

[1] https://www.nytimes.com/2018/03/19/technology/uber-driverles...

[2] https://www.popularmechanics.com/technology/infrastructure/a...

[3] https://www.aviationcv.com/aviation-blog/2019/shocking-facts...

Re: After 6 months of working fine, Tesla software update drives at barriers again

#332

Post by salawat » Thu, Mar 21, 2019, 4:51 PM UTC

Scoundreller wrote: post #123

tptacek wrote: post #119

Earlier quoted context omitted.

Don't Teslas fare significantly worse than cars in similar class in terms of driver fatalities?

Dunno, maybe. I do wonder which cars Tesla gets lumped in with. There can be a difference between a $75k ICE driver and a $75k electric driver. The point is, one stupid decision by a machine doesn’t prove much.

So... Two planes falling out of the sky, killing 400 shouldn't be taken as proving anything?

Or how about 1 stupid decision by a radiotherapy machine? Ever heard of THERAC-25?

In each case, it was just 1 stupid decision by a machine.

The entire reason Engineering as a practice is a thing is because when you implement the capacity for a stupid decision into a system that is then mass produced, dire consequences can result.

I look down on any thought process that doesn't discriminate between the difference between 0 and 1.

If the system provably worked, that decision would not have happened (0). The stupid decision happened (1), however, which means it can happen again at a poorly understood confluence of circumstances.

To err is human, and we forgive each other every day for it.

To err as a machine is a condemnation to the refuse bucket, repair shop, or back to the drawing board.

To err so egregiously as a machine to cause an operator and those around them to lose their life is willful and moral negligence on the part of the system's designer. Slack is cut when good faith is demonstrated, but liability is unambiguous. The hazard would not be there if you hadn't put it there.

Re: After 6 months of working fine, Tesla software update drives at barriers again

#333

Post by darkpuma » Thu, Mar 21, 2019, 5:44 PM UTC

ec109685 wrote: post #273

dmitrygr wrote: post #223

Earlier quoted context omitted.

not if they are on the same axis. then anything along that axis cannot have its depth determined

It can through other signals. You can drive successfully with one eye.

I've got one eye at 20-20 vision, and the second legally blind without correction. My drivers license has a little note that it's not legal for me to drive without my glasses, which I never wear under any other circumstances.

So it's not so clear cut as you make it out to be.

(And you know what? Even if it were legal for me to drive without those glasses, I'd still drive with them. Because ranging is important!)

Re: After 6 months of working fine, Tesla software update drives at barriers again

#334

Post by darkpuma » Thu, Mar 21, 2019, 5:52 PM UTC

tim333 wrote: post #318

tbabb wrote: post #239

Earlier quoted context omitted.

I think a careful interpretation shows how far that is. A self-driving system ought not be considered reliable until it can drive O(100 million miles) without disconnecting once, in order to match human reliability (that's about the distance between fatal accidents currently). A guaranteed disconnect within 25 miles is many zeros of missing reliability. A Disney park engineer once relayed to me the philosophy for des…

I don't know many humans that drive 100 million miles without disconnecting. The disconnect thing is more analogous to pulling over to check the map for which 5000 miles might be ok.

If the computer needs to pull over and check itself into a motel and nap for a few hours, I'm not going to object. But right now the computer "takes a break" by simply disengaging in the middle of the road. Presumably when this happens, the computer isn't even confident in it's ability to safely park the car on the shoulder, as a human driver would attempt in the event of an emergency (blown tire) or undrivable conditions (which for a human might be an intense blizzard or torrential downpour.)

Re: After 6 months of working fine, Tesla software update drives at barriers again

#335

Post by gregknicholson » Thu, Mar 21, 2019, 6:06 PM UTC

vntok wrote: post #303

gregknicholson wrote: post #300

Earlier quoted context omitted.

I wouldn't trust my life to proprietary software, because its makers' motives obviously don't align with my own. If my safety was really paramount, they'd open their software to scrutiny. It's unethical to leave life-and-death decisions to a black-box algorithm, when I know it was written for the primary purpose of gathering more money. Safety is a constraint, not their goal.

You already do that right now and basically have done it every day of your life. The power grid, railway, traffic lights, elevators, etc. all these systems are critical and closed source, and you don't see them killing people on a wide scale nor even on a small scale but regular basis.

Elevators I'll grant you, but the power grid, railway and traffic lights are all controlled by governments, councils and NGO-type organisations — they all have reliability as their primary concern, not churning out units for profit.

And it may be irrational, but wrongly-activating brakes feel like less of a risk than wrongly-activating steering or accelerators.

And anyway, “lots of things are untrustworthy” is not a good argument for trusting something else.

Re: After 6 months of working fine, Tesla software update drives at barriers again

#336

Post by justinv » Thu, Mar 21, 2019, 9:22 PM UTC

Scoundreller wrote: post #85

sbierwagen wrote: post #57

Earlier quoted context omitted.

The section of road shown in the video is the I-5 express lane where it rejoins the regular freeway in North Seattle: https://www.google.com/maps/@47.7021836,-122.3299637,3a,71.1... Note how it's configured for southbound traffic in the street view image. It switches direction twice a day. Though you can't see it very well in the video, there is a crash arresting net after all the signs. https://www.google.com/maps/@…

I followed back the first link for the opposite direction of travel and didn’t find any signage before the swing-gates. You kinda truck along at 45mph+, sticking to the left to continue and with a small amount of notice time, you need to fork to the right. I get that a local would know you are to always fork to the right, but someone unfamiliar or a machine could be confused by the initial half barriers, with only a…

There is signage - there are signs to tell you whether the Express Lanes are open or not that are level with the general signage for direction of travel (ie above the road)

Re: After 6 months of working fine, Tesla software update drives at barriers again

#337

Post by Scoundreller » Thu, Mar 21, 2019, 9:34 PM UTC

salawat wrote: post #332

Scoundreller wrote: post #123

Earlier quoted context omitted.

Dunno, maybe. I do wonder which cars Tesla gets lumped in with. There can be a difference between a $75k ICE driver and a $75k electric driver. The point is, one stupid decision by a machine doesn’t prove much.

So... Two planes falling out of the sky, killing 400 shouldn't be taken as proving anything? Or how about 1 stupid decision by a radiotherapy machine? Ever heard of THERAC-25? In each case, it was just 1 stupid decision by a machine. The entire reason Engineering as a practice is a thing is because when you implement the capacity for a stupid decision into a system that is then mass produced, dire consequences can re…

And people have died and been paralyzed as a direct result of the flu vaccine. Death and paralyses that would not have occurred had they not received that vaccine.

Does that mean the flu vaccine should get dumped in the refuse bin?

Similarly, some other aircraft flying today/tomorrow has automation with an unknown bug/issue that will cause loss of life. Should we disable everything except the 6-pack and stick and rudder?

It would have saved the lives killed by automation, but we would have more aviation death overall.

Re: After 6 months of working fine, Tesla software update drives at barriers again

#338

Post by tbabb » Thu, Mar 21, 2019, 10:06 PM UTC

goshx wrote: post #287

carlivar wrote: post #279

Earlier quoted context omitted.

And if your trip on autopilot is 99.9% reliable, how would you feel about that stat? Do you find yourself paying attention to the road context the entire time? I'm curious if your mental acuity has dropped over time as the car has driven you.

You start trusting it more as you use it and learn its flaws, so you can anticipate when it may do something stupid. I pay attention the entire time, but it is a much more relaxed experience. I’d still pay attention the entire time even if it were supposedly 100%. It is very sad that a lot of people in this forum are hoping this never works. It is one of the most exciting advancements in technology that can benefit u…

> It is very sad that a lot of people in this forum are hoping this never works.

Top level OP here. I am personally rooting for self-driving to succeed and catch on. I just don't think Tesla's current strategy is likely to work, and their cavalier, unjustified overconfidence is either going to sink them or kill people, or both, neither of which is good for the future of self-driving.

Re: After 6 months of working fine, Tesla software update drives at barriers again

#339

Post by m463 » Fri, Mar 22, 2019, 2:20 AM UTC

mprev wrote: post #321

m463 wrote: post #272

Earlier quoted context omitted.

I think this graph: https://upload.wikimedia.org/wikipedia/commons/a/a5/Causes_o... puts a lot of arguments (and young people) to rest.

Not at all. It says that car accidents are a problem. It does not say that the answer is to pretend your driver assistance tech is actually self-driving tech.

I was addressing the point in the parent comment:

> The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans.

Re: After 6 months of working fine, Tesla software update drives at barriers again

#340

Post by goshx » Fri, Mar 22, 2019, 2:40 AM UTC

tbabb wrote: post #338

goshx wrote: post #287

Earlier quoted context omitted.

You start trusting it more as you use it and learn its flaws, so you can anticipate when it may do something stupid. I pay attention the entire time, but it is a much more relaxed experience. I’d still pay attention the entire time even if it were supposedly 100%. It is very sad that a lot of people in this forum are hoping this never works. It is one of the most exciting advancements in technology that can benefit u…

> It is very sad that a lot of people in this forum are hoping this never works. Top level OP here. I am personally rooting for self-driving to succeed and catch on. I just don't think Tesla's current strategy is likely to work, and their cavalier, unjustified overconfidence is either going to sink them or kill people, or both, neither of which is good for the future of self-driving.

We’ll see how this post will age.