Today's Outage Post Mortem
blog.cloudflare.com
Today's Outage Post Mortem
1–10 of 159 posts
Re: Today's Outage Post Mortem
#2Wow, that's pretty fast turnaround for a post-mortem (although it looks to have been a simple problem, so easier to figure out what to write)
Re: Today's Outage Post Mortem
#3I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response.
It appeals to my limited knowledge and non-existant experience that this would be a solution to the prevention of this occurring again in the future?
Re: Today's Outage Post Mortem
#4I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response. It a…
Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've designed that fault tolerance in. This was a problem because the crashes happened system-wide. In the end, we hadn't anticipated that a simple filtering rule like this would cause such a router crash, which was a bad assumption on our part.
Re: Today's Outage Post Mortem
#5I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response. It a…
Good idea. There might be a number of rules that can limit their communication with the routers (and even OOB)
Re: Today's Outage Post Mortem
#6So what are they going to change as a consequence? It seems logical to not rely on a single router vendor anymore, or to test new rules on a staging setup at least for a very short time before pushing them to all routers.
Re: Today's Outage Post Mortem
#7So what are they going to change as a consequence? It seems logical to not rely on a single router vendor anymore, or to test new rules on a staging setup at least for a very short time before pushing them to all routers.
Running Vendor J and Vendor C routers together means you get exposed to the weird bugs in either's open/interoperability code, and lose out on all the advanced features (since most of the good stuff isn't well supported in true cross-platform vendor independent fashion).
It's probably more reasonable to split your network into a few more independent sections and never do updates which affect everything, but unless you're building the space shuttle (and can accept vastly higher costs and lower performance), it's probably better to pick one hardware platform, at least now.
Re: Today's Outage Post Mortem
#8So as far the DOS attack was very successful. It took down the site which it intended to and take down the network with lots and lots of the sites.
Hope lessons are learnt and your next generation is less prone to these attacks
Re: Today's Outage Post Mortem
#9if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability.
e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).
Re: Today's Outage Post Mortem
#10I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response. It a…
Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've de…
Thanks for responding Matthew, and I can imagine the sense of frustration you're all feeling.
My 2cents:
Any change should be considered dangerous, and be tested first, time weighted to it's level of change. (An internal policy that could be communicated publicly. One that I apply to all my staff)
It would be also good to have data center clusters (preferably a datacenter clusters are sharded among regions) which would allow these changes to happen as necessary. A random cluster being the "first cluster" with a roll back in place if fails, or proceed through to other clusters progressively until all are live.
The sharding should hopefully alleviate any corner of the world taking any massive hit due to degraded performance.