Earlier quoted context omitted.
The problem is that there are literally uncountably many situations that a human with "general intelligence" will understand and react to accordingly. Sometimes smoothly, sometimes less so. But a non-conscious automatic entity needs to have the required behavior programmed in explicitly. So yes, you might argue that for this particular situation, you "just" need to put in the proper programming and AI/ML training and…
> needs to have the required behavior programmed in explicitly This is missing the entire point of ML. ML is literally defined as not having to explicitly program responses in for every situation. Needing to pull over because a fire truck has told you it's coming your way in 2 minutes is pretty easy compared with some of those other "uncountably many" situations these cars need to deal with. > the autonomous machine…
So how do you know how the system will respond to an arbitrary situation? You could easily argue that we don't know how an arbitrary human will respond to an arbitrary situation, but we have systems in place to deal with the consequences if they handle it badly.
For example, if a driver handles a situation badly enough, they could lose their license. If an autonomous car does something bad enough that a human would have lost their license, what happens? Do all of that company's cars get pulled off the road until the bug is fixed and validated?