Of course, a counterpoint is what's been happening in aviation. Autopilot became a thing. Autoland became a thing. And, to keep improving planes (first military planes, then commercial aircraft), it was much easier to drop the mechanical connection to the wings.
Autopilot started as a help to pilots, and evolved to something that is a necessity and pilot control inputs are "suggestions" or "goals", not inputs like turning the wheel on a bike. To be followed in what you might refer to as "the long term" from the perspective of controlling the aircraft, but in the short term, the computer is to fly the plane in a way IT thinks is reasonable. An extreme example would be to enforce the flight envelope. But today there exist autoland-only airports (as well as huge airports that go autoland-only if things are too hard for humans, like LHR)
Most of today's passenger aircraft cannot be flown if fly-by-wire is not operational. Most of today's aircraft actually used for passenger transport cannot land without fly-by-wire.
A number of military aircraft, and rocket planes and rockets, even the ones carrying humans, and more and more passenger planes cannot be flown by humans, not just because the mechanical force humans can generate cannot move the control surfaces (which "can be fixed" with hydraulics, if you don't mind serious caveats), but because the human brain is incapable of generating sufficient control inputs at a fast enough rate, or just can't keep stable flight going.
Hilariously, this also goes for hobby quadcopters. They are flown by algorithms. Humans can't do it. Not fast enough. Humans provide direction. Algorithms, even AI algorithms that aren't even guaranteed to succeed at all (in professional/military drones), actually fly the thing.
But, yes, you're entirely correct by saying "then there are lots of new ways to fail". It also works better, cheaper, faster, safer, more comfortable, ... if it doesn't fail.
And ... robotaxis are already far safer than even a good human driver. So whatever the problems ... they don't actually make things worse.
Also you should check out geohot's business. A lot of cars already are "fly-by-wire". Their solution? They now have 2 CAN buses instead of one. One for the critical stuff. Cylinder timings. Checking the oil levels. Turning the wheels. Actuating the brakes. That sort of stuff. A second CAN bus for your bluetooth music, and displays and what have you. I hear a certain new Mercedes now has like 7 buses. We are making things safer.
We can make this work. We will make this work.