Earlier quoted context omitted.
Not mission critical, rather life critical. I've worked on several life critical systems as a security researcher, helping to develop controls in tightly controlled environments to keep people safe. Work on software controlled switches in industrial machines mostly. As I've followed these developments and those of Google and Tesla I've been wondering at times if I live in some alternate universe. The level of diligen…
If 10,000 people were killed on the way to making driverless cars safe, and this wild west approach ended up getting them into mass market use a few years earlier, it would save the lives of thousands of people overall. Is this a bad thing? Is it not justified by utilitarianism?
See, we already have cars, it's not like we're going from 'horses and buggies' to cars, it is going from one good solution to an even better one and even though there is a risk at being stuck at the present local maximum for a little longer going down in order to find the peak of this particular mount improbable would not be an acceptable path for the majority of the public.
And without having a time machine or a functional crystal ball you won't know if it is 10K, 100K or a few million people that get killed before you get it right so the bar has been set at 'it needs to be better than the present level before it is acceptable'.