This reminds me of a presentation by Twitter at a conference where they discussed MTBR (Mean Time Before Recovery) instead of MTBF (Mean Time Before Failure). Instead of trying to prevent all possible failures, they focused on acceptable failure and speedy recovery. If a server takes twice as long to response to a request, mark it offline and let the automated recovery process diagnose the problem and if it's just an…
A great example of this was that Bentley used the lightest bolt they could everywhere on the car to minimize weight, but Audi used the same size bolt everywhere to minimize the number of tool changes needed to replace the parts.