Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will your self-driving car be programmed to kill you?

uab.edu

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Will your self-driving car be programmed to kill you?

#51

Post by nhdev »

nhdev wrote: post #35
_lce0 wrote: post #33

I can imagine this scenario: in a bridge without any place to go but ahead .. ------------------------------------------------- 5 people a) The car can break, but the heavy truck below will certainly kill the car's only passenger, who's on the back seat b) The car hit and potentially kill most of the people standing, but save its passenger c) The car can jump to the void Scary as it sounds, what the car would do depe…

(d) The car would stop because unlike human drivers, a self-driving car would be programmed to avoid going so fast that it can't stop if a sudden obstacle would appear (aside from maybe something falling from the sky).

A human would drive too fast on in icy conditions... if the computer knows there are icy conditions and it doesn't slow down before it even senses trouble then the car was programmed to be going too fast. If it is so dangerous there is no way to remove these scenarios (like a blizzard) then a human should be forced to override the system in which case the human is at fault.

I trust sensors to detect icy conditions better than a trust myself.

[edit] Most bridges where I am have signs that explicitly warn that bridges freeze over. And people intuitively know... bridge + recent precipitation + cold whether = slow down. I don't know why a computer wouldn't know that.

Re: Will your self-driving car be programmed to kill you?

#52

Post by yk »

kailuowang wrote: post #17
pestaa wrote: post #7

Earlier quoted context omitted.

So when humans mount horses and go into the woods to hunt as entertainment, they are not superior either?

Computer may have the same intelligence as human, but their fundamental inner drive will be completely different. Human like any other animal has a deep genetically programmed drive: survive and reproduce. Hunting as entertainment, for example, is actually practicing for survival - all hunting animals do that. This deep drive is the result of evolution. AI on the other hand has a completely different evolution proces…

That is in no way certain. In the case where a AI is the result of a evolutionary algorithm for example, there is a distinct possibility that the AI has necessarily a taste for competition. Or if important parts of the AI are modeled on the human brain, then it is entirely possible that a AI is closer to a modified human.

Re: Will your self-driving car be programmed to kill you?

#53

Post by mnw21cam »

_lce0 wrote: post #33

I can imagine this scenario: in a bridge without any place to go but ahead .. ------------------------------------------------- 5 people a) The car can break, but the heavy truck below will certainly kill the car's only passenger, who's on the back seat b) The car hit and potentially kill most of the people standing, but save its passenger c) The car can jump to the void Scary as it sounds, what the car would do depe…

I don't think a car that breaks whenever placed in a challenging position is going to sell very well.

One that brakes, well that's different.

Re: Will your self-driving car be programmed to kill you?

#54

Post by baldfat »

pjkundert wrote: post #46

Remember: we do not even have cruise control that is usable in the winter. Once any autonomous driving system can provide even this basic function (not soon, I fear), then we can talk about more complex behavior... This debate about what autonomous vehicles will do in extreme situations is but the "tip of the iceberg". Until all autonomous drive systems can be "unit tested" under a standard suite of simulated sensor…

I don't think slippery is an issue for computers. We have the quick on off brakes of ABS Braking. We also have AWD of Subaru for a decade that can transfer power to other wheels if it starts to slip. Also it can steer into a slid quicker and more precise than 99% of people right now.

Re: Will your self-driving car be programmed to kill you?

#55

Post by drzaiusapelord »

pjkundert wrote: post #46

Remember: we do not even have cruise control that is usable in the winter. Once any autonomous driving system can provide even this basic function (not soon, I fear), then we can talk about more complex behavior... This debate about what autonomous vehicles will do in extreme situations is but the "tip of the iceberg". Until all autonomous drive systems can be "unit tested" under a standard suite of simulated sensor…

This is a great point. Autonomous driving is very primitive right now and the idea of an "all knowing" computer judging our lives is ridiculous. It'll react the same way we react, with dumb reflexes. If an obstacle appears on the road, it will brake or swerve away from the target. It won't have time to weigh some massive AI fuzzy logic about who dies. It will have no idea what the survivability of any of this is, and like a human it will try to avoid the collision.

It seems that 60's AI is back in fashion and now part of autonomous car culture. There's this faulty idea of this super brain driving these cars. The reality is we're looking at dumb as rocks firmwares that can't even drive in the snow or heavy rain properly. Frankly, I think these firmwares should stay dumb like the human reflex system. Too much "engineer logic" and other questionable heuristics will just lead to bugs, lag, etc. I don't want my car to drive off the cliff because of some software bug. I don't need some Samsung engineer on a time constraint who couldn't spare the cycles for proper edge detection because he was wasting those cycles on 'sport mode' which marketing made a big fuss over delivering this quarter. I want dumb and simple code to do what's a dumb and simple task.

Re: Will your self-driving car be programmed to kill you?

#56

Post by judemelancon »

baldfat wrote: post #49
_lce0 wrote: post #33

I can imagine this scenario: in a bridge without any place to go but ahead .. ------------------------------------------------- 5 people a) The car can break, but the heavy truck below will certainly kill the car's only passenger, who's on the back seat b) The car hit and potentially kill most of the people standing, but save its passenger c) The car can jump to the void Scary as it sounds, what the car would do depe…

In 4/28/2006 Radio Lab (PBS Amazing Podcast) did a show on morality. They tell the moral choice people will make: The program begins with a moral conundrum. You're working with a crew doing track repairs (presumably, using heavy equipment such that you can't hear much else) when you happen to look up to see a locomotive rushing towards five of your buddies who are working on the tracks. But just as quickly you realiz…

I honestly think that apparent flip-flop comes down entirely to how terribly ineffective the large human sounds at stopping the train. Telling people to pretend it works is not an effective measure. People can't cancel their biases just from being told to do so, and a bias against such an awful plan is quite reasonable.

Re: Will your self-driving car be programmed to kill you?

#57

Post by pjkundert »

baldfat wrote: post #54
pjkundert wrote: post #46

Remember: we do not even have cruise control that is usable in the winter. Once any autonomous driving system can provide even this basic function (not soon, I fear), then we can talk about more complex behavior... This debate about what autonomous vehicles will do in extreme situations is but the "tip of the iceberg". Until all autonomous drive systems can be "unit tested" under a standard suite of simulated sensor…

I don't think slippery is an issue for computers. We have the quick on off brakes of ABS Braking. We also have AWD of Subaru for a decade that can transfer power to other wheels if it starts to slip. Also it can steer into a slid quicker and more precise than 99% of people right now.

You are quite correct.

The main problem right now is -- there is no way for you to know how any particular autonomous driving system will handle it. With every second auto maker and start-up claiming to have independently developed super-duper auto-drive technology, I think it will get worse before it gets better.

Without standardised testing frameworks (ie. sending my car's autonomous driving system through a simulator to see what it will do), I remain ... unconvinced.

Being an industrial programmer for 30 years, I think I'll continue to err on the side of assuming incompetence and failure, until proven otherwise.

Re: Will your self-driving car be programmed to kill you?

#58

Post by eli_gottlieb »

gambiting wrote: post #25
imgabe wrote: post #14

It's an interesting thought experiment, but I think a real situation with such a stark binary choice is so improbable that it's barely worth considering. Cars are not on trolley tracks and computers can brake or swerve harder and faster than any human could.

No matter how fast they break or swerve, it's not improbable that they might be in a situation where the computer has a choice - hit someone standing on the side of the road, or don't hit them,but potentially kill everyone on board of the vehicle. What should it do? Would anyone of us want to be the programmer who writes that code? Because I know that I definitely wouldn't want to be in charge of that.

Well the issue is, we need to program the cars so that we optimize the outcome when every car on the road behaves that way and is expected to behave that way.

Do we get the best outcome when everyone expects the computer to spare the pedestrian? When everyone expects the computer to drive "selfishly" or to try to minimize its own impact velocity?

I don't think it's a good idea to program the cars with a "utilitarian" policy of saving the most human lives possible, as not only does that require quite a lot of inferential power to be packed into the car, it doesn't set up any concrete expectations for the passengers and pedestrians. You would have to add a mechanism for the car to first, in bounded real-time, infer how to save the most lives, and then signal to all passengers and bystanders exactly what course of action it has chosen, so that they can avoid acting randomly and getting more people killed.

This is why we always demand very regimented behavior in emergency procedures: trying to behave "smartly" in an emergency situation, in ways that don't coordinate well with others, actually gets more people dead. Following a policy may "feel" less "ethical" or "heroic", but it actually does save the most lives over the long run.

Re: Will your self-driving car be programmed to kill you?

#59

Post by zaphar »

rijncur wrote: post #23

I (perhaps somewhat naively) assumed that a self-driving car would be programmed to prioritise the occupants of the vehicle (as that most closely emulates the likely reactions of a human driver in that situation). Self-driving cars are likely to be safer than human drivers anyway, so one has to consider how much risk there is of a situation like the Trolley Problem arising - i.e. not much.

So here's the thing. The first time that an accident kills pedestrians with a self driving car involved the creator of that car will be open to a lawsuit. And in court the question of what that car should have been programmed to prioritize will come up. A response of we tried to emulate the likely reactions of a human driver in this situation won't hold up. The whole point of a self driving car as you said is that they are safer. In fact safer to the community as a whole. So if a car plows into crowd in order to avoid colliding with an obstacle and killing the rider. (something a real human would likely due without even realizing it) Would be grounds for that suit to award damages to the creator of the cars software.

It's not as simple as it sounds at first.

Re: Will your self-driving car be programmed to kill you?

#60

Post by netcan »

stkni wrote: post #31

Don't think we're going to ask computers to count the cost of a life, anytime soon. Especially since humans have a hard enough job doing it [1]. It will be designed, I imagine, to do the safest thing possible given an emergency. Which is in this instance, as most people suggest, stop as quickly as possible. Nothing to see, move on. [1] http://www.nytimes.com/2011/02/17/business/economy/17regulat...

I totally agree, but…. :)

As silly and pointless as it to worry about whether robocar morality should be primarily deontology or utilitarian, I have to admit I love it. I'm a sucker for this kind of science fiction thinking, brought on by everyone's expectations of technological progress. The average person has a tremendous amount of confidence of the "they'll figure it out" kind.

I think it's a good sign, the opposite of cynicism. Also, I think moral questions are interesting and if teaching is the best way to learn, maybe imagining how to teach a machine to be moral is a good exercise.

On that note, deontology fits nicely with the way computers tend to think - rules. OTOH, utilitarianism means seeing the future - a nice use for all our future computational power. While we're on that, do computers have a duty to be true to their nature?

Post reply on HN