Tesla's self driving algorithm's overlay [video]

An example from my own backyard. My car is equipped only with ACC and lane assist. I already had a couple of situations where in severe rain the sensors are no longer able to „see”. The car throws a fit and I have to take over immediately. These things are not fit for purpose. They are not able to judge „a big surface covered with water, risk of aquaplaning” or „surface covered with snow, temp dropping, possibility of black ice” or even simple mud / sand.

Maybe if one lives in Nevada, Arizona, California, where it’s always sunny, rains once in ten years, one can get swayed into believing that these things work. However, rsin, dnow, black ice or a mix of those and it becomes very quickly obvious it does not.

These cars, today, they are just dumb machnines without judgement. Never I am going to sit in an autonomous car outside of a controlled environment like an airport / strict city centre. Outside of the city, no way.

By normal standards, it's insane to expect non-experts to maintain and operate heavy machinery, in public streets, half asleep or half paying attention.

It's also a pretty absurd way of providing mass transport in most places.

So if you're going to throw out normal risk assessment as a starting point, where do you go from there? People know how bad quality software can be, how ubiquitous bugs are, and how often programs are deep morasses of strange and perverse hacks, and how it only doesn't (normally) kill people because the stakes are small. Then Uber's car actually did kill someone, and predictably, the familiar problem of bad, buggy software looms larger, as a risk, than the dull and more or less commonly accepted fact of cars hitting people all the time.

Very common threats, like pneunomia, poverty, or cancer, tend to get depreciated in comparison to symbolic, unusal threats - and I think it's pretty unsurprising that on HN, people feel threatened by the possibility of bugs not only ruining their day, but actually running them over.

I wonder if the fear has more to do with it simply being different. The set of things that a CV system misses is likely quite different from the set of things that a human misses. A CV system is likely to make mistakes that a human would never make, and even if those mistakes are far fewer in number, they make it look bad to humans.

We've all seen electronic devices do this. When an Amazon Echo interprets the sound of a tea kettle as its wake word, we're left with no explanation, and no way to rationalize it other than to assume the device is, in some way, bad at its job. Surely a human would never make this mistake, we think, and so therefore the device must be dumber than a human at word detection. And yet, there surely exists a class of noises that a human would mistake for a word that an Echo would not - but that doesn't factor in.

Anyway, this means that a CV-controlled car would behave, compared to a human, strangely. When one's intuition about the car's behavior, tuned from years of experience both driving and observing cars, differs from the car's actual behavior, it becomes harder to make predictions about how the car will behave, and from here comes a fear and a sense of uneasiness. And when it makes a rare mistake that a human would never make, because it's perceptual system is entirely different, we assume its dumber than a human all-together.

    Patch for car stopping way too often:
        catch (NotUnderstoodException)
        {  
    -        SlowDownAndStopLikeAnyoneElseWould();
    +        // FIXME TODO let the safety driver handle it for now
    +        // SlowDownAndStopLikeAnyoneElseWould();
        }