Earlier quoted context omitted.
> it's been working so well I wonder if they'd even notice before it's too late I guess the solution is to simulate faults everywhere at all times in production. Something most systems definitely ignore. Fault tolerance is just too different from "normal" way of writing software, it can't be done well without paradigm shift.
You were downvoted but I think you're right. Netflix's Chaos Monkey is a similar idea: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
I mean imagine if you have a distributed system like an object storage or a database. Before deploying it into production you try to evaluate how it performs in various conditions. But you can't really do it thoroughly enough and on a lot of data, you can only scratch the surface and this forms your expectations about performance. You deploy it into production and start monitoring how it performs and relying on that performance that you barely understand. Everything seems fine, until you add nodes to the system or replace an hdd and suddenly it becomes so slow, that nothing works anymore. Turns out everyone, including system designers, had broken assumptions about its performance and didn't even think about it, because testing fault tolerance was always done only separately from normal mode of operation and it was impossible to grasp how they affect each other and come up with better algorithms.