They said they ran a dry-run. What did that do, just generate these diffs? I would have expected them to have some way of simulating the network for BGP changes in order to verify that they didn't just fuck up their traffic.
Cloudflare outage on June 21, 2022
151–160 of 234 posts
Re: Cloudflare outage on June 21, 2022
#152In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!
Well cloudflare’s entire value is in uptime and preventing outages. Showing they have a rapid response and strong fundamental technical understanding is much more critical in the “prevent downtime” business.
Re: Cloudflare outage on June 21, 2022
#153Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?
You can learn BGP with mininet: https://mininet.org/ You can simulate arbitrarily large networks and internetworks with this, provided you have the hardware to run a large enough number of virtual appliances, but they are pretty lightweight.
Mininet is what the Georgia Tech OMSCS Computer Networking labs use. It's not bad, the two labs that stood out to me were using it to implement BGP and a Distance Vector Routing protocol.
Re: Cloudflare outage on June 21, 2022
#154Something else that I think would be smart to implement is a reorder detection. Have the change approval specificy point out stuff that gets reordered, and require manual approval for each section that gets moved around. I also think that having a script that walks through the file and points out any ovibious mistakes would be good to have as well.
Yeah, there's got to be some sweet spot between "formally verify all the things" and "i guess this diff looks okay, yolo!".
I'd say that if you're designing a system which has the potential to disconnect half your customers based on a misconfiguration, then you should spend at least an hour thinking about what sorts of misconfigurations are possible, and how you could prevent or mitigate them.
The cost-benefit analysis of "how likely is it such a mistake would get to production (and what would that cost us)?" vs "how much effort would it take to write and maintain a verifier that prevents this mistake?" should then be fairly easy to estimate with sufficient accuracy.
Re: Cloudflare outage on June 21, 2022
#155What's it like to be an engineer designing and working on these systems? Must be sooo fulfiling! #Goals; Y'all are my heores!!
https://www.cloudflare.com/careers/
I'm currently waiting on a recruiter to get my panel interviews scheduled. You guys are in "dream gig" territory for me. Any tips? ;-)
Re: Cloudflare outage on June 21, 2022
#156In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!
No provider is perfect, but it's because of stuff like this that I trust Cloudflare waaaaaaaaaaay more than the likes of Amazon. Transparency engenders trust, and eventually, love! Thank you, Cloudflare.
The sheer level of technical competence of your engineering team continues to astound me. (Yes, they made a mistake and didn't catch an error in the diff. But your response process went exactly as it should, and your postmortem is excellent.) I couldn't even begin to think about designing or implementing something of this complexity, much less being able to explain it to a layperson after a failure. It is really impressive, and I hope you will continue to do so into the future!
Most of the companies I've worked for unfortunately don't use your services, but I've always been a staunch advocate and converted a few. Maybe the higher-ups only see downtime and name recognition (i.e. you're not Amazon), but for what it's worth, us devs down the ladder definitely notice your transparency and communications, and it means the world. I've learned to structure my own postmortems after yours, and it's really aided in internal communications.
Thank you again. I can't wait for the day I get to work in a fully-Cloudflare stack :)
Re: Cloudflare outage on June 21, 2022
#157Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-…
You say that like it hasn't been going on since the mid 1990's, when it got deployed.
I'm not blaming BGP, since it prevents far more outages than it causes, but BGP-based outages have been a thing since its beginning. And any other protocol would have outages too - BGP just happens to be the protocol being used.
Re: Cloudflare outage on June 21, 2022
#158Feels a little disingenuous to use the first 3/4 of the report to advertise.
Re: Cloudflare outage on June 21, 2022
#159Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-…
> since engineers got locked out of their systems Sounds like the same happened here: "Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations." But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cl…
> the solution long ago was always a modem on an auxiliary port
Now you can use mobile Internet (4G/5G)
Re: Cloudflare outage on June 21, 2022
#160Earlier quoted context omitted.
Thats like blaming the hammer for breaking. BGP is just a tool, it would be something else to do the same purpose.
Some tools are more fragile and error prone than others.
Except that this wasn't an example of BGP being prone to error or fragile. This was, as the blog post specifically calls out, human error. They put two BGP announcement rules after the "deny everything not previously allowed" rule. It's the same as if someone did this to a set of ACLs on a firewall.
The main difference between BGP and all other tools is that if you mess up BGP, you've done a very visible thing because BGP underpins how we get to each other's networks. But it's not a sign of BGP being fragile, just very important.