I am just going by the outage post mortem report. I could not read the article after I read the first few lines - "feature file". I am stuck at consuming the design idea here where you allow multiple inserts for one feature [assuming you have some uniqueness constraint]. Even a simple key-value map per feature should have made the insertions as just a put/replace the value. I think that was not the case here, where cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here cloudflare should rethink. If someone can educate me on the design, I can continue reading the next lines.
Cloudflare outage on November 18, 2025 post mortem
891–900 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#892I am just going by the outage post-mortem report. I could not read the article after I read the first few lines - "feature file" expansion/limits. I am stuck at consuming the design idea here, where you allow multiple inserts for one feature [assuming you have some uniqueness constraint].
Even a simple key-value map per feature should have allowed for insertions as simple as a put/replace of the value and not appending to the file. That was not the case here, where Cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here that Cloudflare should rethink. If someone can educate me on the design, I can continue reading the next few lines.
Re: Cloudflare outage on November 18, 2025 post mortem
#893Earlier quoted context omitted.
Thanks for the explanation! This definitely reminds me of CrowdStrike outages last year: - A product depends on frequent configuration updates to defend against attackers. - A bad data file is pushed into production. - The system is unable to easily/automatically recover from bad data files. (The CrowdStrike outages were quite a bit worse though, since it took down the entire computer and remediation required manual…
It might remind you of Crowdstrike because of the scale. Outages are in a large majority of cases caused by change, either deployments of new versions or configuration changes.
zone your deployments first -blue/green. Have a small blue zone, and test it out. If it works, then expand to green deployments.
A configuration file should not grow! design failure here, I want to understand
Re: Cloudflare outage on November 18, 2025 post mortem
#894Earlier quoted context omitted.
Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code. And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.
I would love to go further and explicitely forbid unwrap and similar calls using a `no_panic` attribute.
I actually have to do this for programs that runs in bare metal. You can't afford to have nondeterministic panic like this. If things really gone wrong you'd have a watchdog and health checker to verify the state of program.
Re: Cloudflare outage on November 18, 2025 post mortem
#895Thanks for the detailed RCA. After going through the blog post as well, there’s a curiosity about whether there are opportunities to detect such scenarios earlier in the testing phase. It would be helpful to understand if there are any differences between the test environment and production that might have contributed to this. This could provide insights into strengthening the process going forward.
Re: Cloudflare outage on November 18, 2025 post mortem
#896Earlier quoted context omitted.
Sure, but the same is true of any error handling strategy. When you work with exceptions, the key is to assume that every line can throw unless proven otherwise, which in practice means almost all lines of code can throw. Once you adopt that mental model, things get easier.
Explicit error handling strategies allow you to not worry about all the code paths that explicitly cannot throw -- which is a lot of them. It makes life a lot easier in the non-throwing case, and doesn't complicate life any more in the throwing case as compared to exception-based error handling. It also makes errors part of the API contract, which is where they belong, because they are.
I would respectfully disagree with most of what you said. I guess individual perspective depends a lot on the kinds of code you work on.
The point about being explicitly part of the API stands, though.
Re: Cloudflare outage on November 18, 2025 post mortem
#897Earlier quoted context omitted.
That just means it takes longer to test. It may not be possible to do it in a reasonable timeframe with the volumes involved, but if you already have 100k servers running to serve 25M requests per second, maybe briefly booting up another 100k isn’t going to be the end of the world? Either way, you don’t need to do it on every commit, just often enough that you catch these kinds of issues before they go to prod.
> maybe briefly booting up another 100k isn’t going to be the end of the world Cloudflare doesn’t run in AWS. They are a cloud provider themselves and mostly run on bare metal. Where would these extra 100k physical servers come from?
From their desire to representatively test before they deploy to production?
Doing stuff at scale doesn’t suddenly mean you skip testing.
And just because they host stuff themselves doesn’t mean they couldn’t run on the cloud if they needed to.
Re: Cloudflare outage on November 18, 2025 post mortem
#898Earlier quoted context omitted.
> Returning a Result by definition means the method can fail. No more than returning an int by definition means the method can return -2.
> No more than returning an int by definition means the method can return -2. What? Returning an int does in fact mean that the method can return -2. I have no idea what your argument is with this, because you seem to be disagreeing with the person while actually agreeing with them.
> What? Returning an int does in fact mean that the method can return -2.
What? No it doesn't.
fn square(n: i32) -> i32 {
n * n
}
Though in this case it's more like knowing that the specific way you call the function in foo.rs will never get back a -2.
fn bar(n: i32, allow_negative: bool) -> i32 {
let new = n * 2;
if allow_negative || new >= 0 { new } else { 0 }
}
bar(x, false)Re: Cloudflare outage on November 18, 2025 post mortem
#899[dead]
Re: Cloudflare outage on November 18, 2025 post mortem
#900Earlier quoted context omitted.
This wasn't a runtime property that could not be validated at compile time. And you don't need to fall back on "OS level security and reliability" when your type system is enforcing an application-level invariants. In fact I'd argue that crashing is bad. It means you failed to properly enumerate and express your invariants, hit an unanticipated state, and thus had to fail in a way that requires you to give up and fal…
> The end result was a massive world-wide outage. The world wide outage was actually caused by deploying several incorrect programs in an incorrect system. The root one was actually a bad query as outlined in the article. Let’s get philosophical for a second. Programs WILL be written incorrectly - you will deploy to production something that can’t possibly work. What should you do with a program that can’t work? Pret…
> Programs WILL be written incorrectly - you will deploy to production something that can’t possibly work. What should you do with a program that can’t work? Pretend this can’t happen? Or let you know so you can fix it?
Type systems provide compile time guarantees of correctness such that systems cannot fail in ways covered by the type system.
In this case, they used an unsound hole in the type system to do something that unnecessarily abandoned those compile-time invariants and in the process caused a world-wide outage.
The answer is not to embrace poking unsound holes in your type system in the first place.