> Let this be a lesson to not introduce single points of failure into critical systems without having prepared for their unavailaility or misbehaviour.
I think that is where this is going to get more complicated, I think that this has broadened what we traditionally think about when it comes to what we consider a single point of failure.
I would wager that most people would not have considered something like Crowdstrike to be a single point of failure. It is not what you would generally think of.
From what sounds of the bits and pieces of Delta that came out, it sounds like the biggest issue they had was that so many of these systems were not together. It required physical access to systems that were in many different locations (I would love to be corrected on this). Under normal circumstances that would cover you in the case of an outage in a particular location. If you built your system to handle entire areas going down, adding in redundancy.
But this was very much a unique situation. We really shouldn't pretend otherwise. It wasn't an external service going down, or AWS going down, loss of internet connection, or other things going down that we would normally account for in disaster recovery. Hell I would argue that in this particular situation, you could have the best disaster recovery but it wouldn't have done anything since Crowdstrike was probably baked into your images. So have fun bringing up an instance that was going to instantly brick. Yeah eventually that would have been a non issue, but I can not imagine a scenario that I would have ever thought that I had a baked image and assuming I still had access to it and I could assure that it was the image that I made, that it somehow would itself be a problem.
Before a couple weeks ago if we had built a system that was redundant enough to handle parts going down, different geographical locations, etc etc with, maybe things slow down but still works. All of the standard things that we talk about.
It isn't absurd to think that your entire system going down in a situation where you don't think you have a single point of failure, is nearly impossible or if it happens there is something outside of your company seriously bad.