> The worst man-made catastrophes seem to frequently involve a human manually disabling some sort of critical safety mechanism (eg, Chernobyl).
But again, we're back to perverse incentives. Keeping the plant running was more important than diagnosing and fixing any problems. If the people involved were first assessed by how well they found and fixed problems, and then by how much uptime the plant had, I'm confident it would have played out quite differently.
There is a point to what you're saying though, and how it interacts with my statement. Humans work based on current incentives, while machines work towards the goals their designers were incentivized towards at the time of their creation. Keeping the former in good order over time is a harder task than making sure the latter is correct at design time. That is, it's easy in the design process to emphasize the cost of errors and try to account for them, but humans managers, being more dynamic, sometimes allow perverse incentives to change how a human workforce behaves accidentally. Human workers are more versatile, but that versatile comes with downsides.
> In theory, automation could encompass the knowledge of the finest experts in the field
Which is why I qualified my statement with "until every mistake that can happen is known and accounted for" and "allow for". The point is, how many machines are operating in an environment where that's true, and we're actually utilizing the information from all the experts correctly? Just taking into consideration that the physical environments where machines run is often not fully under control, I think it's actually rather small.