If the software has a limit on the size of the feature file then the process that propagates the file should probably validate the size before propagating ..
Cloudflare outage on November 18, 2025 post mortem
571–580 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#572This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
> This is the multi-million dollar .unwrap() story. While there are certainly many things to admire about Rust, this is why I prefer Golang's "noisy" error handling. In golang that would be either: feature_values, err := features.append_with_names(...) And the compiler would have complained that this value of `err` was unused; or you'd write: feature_values, _ := features.append_with_names(...) And it would be far mo…
I haven't been writing Rust for that long (about 2 years) but every time I see .unwrap() I read it as 'panic in production'. Clippy needs to have harder checks on unwrap.
Re: Cloudflare outage on November 18, 2025 post mortem
#573A fucking unhandled exception brought down a majority of the internet? Why do we continue to let these clowns run a large portion of the internet?
Big tech is a fucking joke.
Re: Cloudflare outage on November 18, 2025 post mortem
#574Earlier quoted context omitted.
I think defence against a DDOS against your network is the best reason for a quick rollout
Maybe, but in that case maybe have some special casing logic to detect that yes indeed we're under a massive DDOS at this very moment, do a rapid rollout of this thing that will mitigate said DDOS. Otherwise use the default slower one? Of course, this is all so easy to say after the fact..
It's literally in the blog post as well
> In the internal incident chat room, we were concerned that this might be the continuation of the recent spate of high volume Aisuru DDoS attacks:
Re: Cloudflare outage on November 18, 2025 post mortem
#575Earlier quoted context omitted.
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…
They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…
I don't think these are realistic requirements for any engineered system to be honest. Realistic is to have contingencies for such cases, which are simply errors.
But the case for Cloudflare here is complicated. Every engineer is very free to make a better system though.
Re: Cloudflare outage on November 18, 2025 post mortem
#576This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…
I don't understand why they didn't validate and sanitize the new config file revision.
If bad(whatever that reason is) throw an error and revert back to previous version. You don't need to take down the whole internet for that.
Re: Cloudflare outage on November 18, 2025 post mortem
#577Earlier quoted context omitted.
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…
They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…
> Why do they do non-critical changes in production before testing in a stage environment?
I guess the noncritical change here was the change to the database? My experience has been a lot of teams do a poor job having a faithful replica of databases in stage environments to expose this type of issue.
Re: Cloudflare outage on November 18, 2025 post mortem
#578Earlier quoted context omitted.
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…
Does their ring based rollout really truly have to be 0->100% in a few seconds? I don’t really buy this requirement. At least make it configurable with a more reasonable default for “routine” changes. E.g. ramping to 100% over 1 hour. As long as that ramp rate is configurable, you can retain the ability to respond fast to attacks by setting the ramp time to a few seconds if you truly think it’s needed in that moment.
The configuration file is updated every five minutes, so clearly they have some past experience where they’ve decided an hour is too long. That said, even a roll out over five minutes can be helpful.
Re: Cloudflare outage on November 18, 2025 post mortem
#579This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…
Classic @devops_borat
"To make error is human. To propagate error to all server in automatic way is #devops"
Re: Cloudflare outage on November 18, 2025 post mortem
#580Timely post mortem. Sucks to have this happened, but at least they are quite transparent and detailed in the writeup.