I would have been a bit cheeky and opened with 'It wasn't DNS.'
Cloudflare outage on November 18, 2025 post mortem
161–170 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#162This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
To be fair, this failed in the non-rust path too because the bot management returned that all traffic was a bot. But yes, FL2 needs to catch panics from individual components but I’m not sure if failing open is necessarily that much better (it was in this case but the next incident could easily be the result of failing open).
But more generally you could catch the panic at the FL2 layer to make that decision intentional - missing logic at that layer IMHO.
Re: Cloudflare outage on November 18, 2025 post mortem
#163Earlier quoted context omitted.
At Facebook they name certain "escape hatch" functions in a way that inescapably make them look like a GIANT EYESORE. Stuff like DANGEROUSLY_CAST_THIS_TO_THAT, or INVOKE_SUPER_EXPENSIVE_ACTION_SEE_YOU_ON_CODE_REVIEW. This really drives home the point that such things must not be used except in rare extraordinary cases. If unwrap() were named UNWRAP_OR_PANIC(), it would be used much less glibly. Even more, I wish ther…
right and if the language designers named it UNWRAP_OR_PANIC() then people would rightfully be asking why on earth we can't just use a try-catch around code and have an easier life
Probably not, since errors as values are way better than exceptions.
Re: Cloudflare outage on November 18, 2025 post mortem
#164So, to recap: - Their database permissions changed unexpectedly (??) - This caused a 'feature file' to be changed in an unusual way (?!) - Their SQL query made assumptions about the database; their permissions change thus resulted in queries getting additional results, permitted by the query - Changes were propagated to production servers which then crashed those servers (meaning they weren't tested correctly) - They…
Looks like you have the perfect window to disrupt them with a superior product.
just before this outage i was exploring bunnycdn as the idea of cloudflare taking over dns still irks me slightly. there are competitors. but there's a certain amount of scale that cloudflare offers which i think can help performance in general. that said in the past i found cloudflare performance terrible when i was doings lots of testing. they are predominantly a pull based system not a push, so if content isn't current the cache miss performance can be kind of blah. i think their general backhaul paths have improved, but at least from new zealand they used to seem to do worse than hitting a los angeles proxy that then hits origin. (although google was in a similar position before, where both 8.8.8.8 and www.google.co.nz/.com were both faster via los angeles than via normal paths - i think google were doing asia parent, like if testing 8.8.8.8 misses it was super far away). i think now that we have http/3 etc though that performance is a bit simpler to achieve, and that ddos, bot protection is kind of the differentiator, and i think that cloudflare's bot protection may work reasonably well in general?
Re: Cloudflare outage on November 18, 2025 post mortem
#165Earlier quoted context omitted.
It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…
> every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this. How about indexing into a slice/map/vec? Should every `foo[i]` have an infallibility comment? Because they're essentially `get(i).unwrap()`.
Usually you'd want to write almost all your slice or other container iterations with iterators, in a functional style.
For the 5% of cases that are too complex for standard iterators? I never bother justifying why my indexes are correct, but I don't see why not.
You very rarely need SAFETY comments in Rust because almost all the code you write is safe in the first place. The language also gives you the tool to avoid manual iteration (not just for safety, but because it lets the compiler eliminate bounds checks), so it would actually be quite viable to write these comments, since you only need them when you're doing something unusual.
Re: Cloudflare outage on November 18, 2025 post mortem
#166Earlier quoted context omitted.
> when you do unwrap() you're saying the Result should NEVER fail Returning a Result by definition means the method can fail.
> Returning a Result by definition means the method can fail. No more than returning an int by definition means the method can return -2.
What? Results have a limited number of possible error states that are well defined.
Re: Cloudflare outage on November 18, 2025 post mortem
#167Earlier quoted context omitted.
Most recently, a few weeks ago (but you'll find more just a page or two into the blog): https://fly.io/blog/corrosion/
It's great that you're working on regionalization. Yes, it is hard, but 100x harder if you don't start with cellular design in mind. And as I said in the root of the thread, this is a sign that CloudFlare needs to invest in it just like you have been.
I recoil from that last statement not because I have a rooting interest in Cloudflare but because the last several years of working at Fly.io have drilled Richard Cook's "How Complex Systems Fail"† deep into my brain, and what you said runs aground of Cook #18: Failure free operations require experience with failure.
If the exact same thing happens again at Cloudflare, they'll be fair game. But right now I feel people on this thread are doing exactly, precisely, surgically and specifically the thing Richard Cook and the Cook-ites try to get people not to do, which is to see complex system failures as predictable faults with root causes, rather than as part of the process of creating resilient systems.
Re: Cloudflare outage on November 18, 2025 post mortem
#168Earlier quoted context omitted.
What people are saying is that idiomatic prod rust doesn't use unwrap/expect (both of which panic on the "exceptional" arm of the value) --- instead you "match" on the value and kick the can up a layer on the call chain.
What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than…
> What happens to it up the callstack?
as they say in the post, these files get generated every 5 minutes and rolled out across their fleet.
so in this case, the thing farther up the callstack is a "watch for updated files and ingest them" component.
that component, when it receives the error, can simply continue using the existing file it loaded 5 minutes earlier.
and then it can increment a Prometheus metric (or similar) representing "count of errors from attempting to load the definition file". that metric should be zero in normal conditions, so it's easy to write an alert rule to notify the appropriate team that the definitions are broken in some way.
that's not a complete solution - in particular it doesn't necessarily solve the problem of needing to scale up the fleet, because freshly-started instances won't have a "previous good" definition file loaded. but it does allow for the existing instances to fail gracefully into a degraded state.
in my experience, on a large enough system, "this could never happen, so if it does it's fine to just crash" is almost always better served by a metric for "count of how many times a thing that could never happen has happened" and a corresponding "that should happen zero times" alert rule.
Re: Cloudflare outage on November 18, 2025 post mortem
#169I don't get why that SQL query was even used in the first place. It seems it fetches feature names at runtime instead of using a static hardcoded schema. Considering this decides the schema of a global config, I don't think the dynamicity is a good idea.
Re: Cloudflare outage on November 18, 2025 post mortem
#170Earlier quoted context omitted.
At Facebook they name certain "escape hatch" functions in a way that inescapably make them look like a GIANT EYESORE. Stuff like DANGEROUSLY_CAST_THIS_TO_THAT, or INVOKE_SUPER_EXPENSIVE_ACTION_SEE_YOU_ON_CODE_REVIEW. This really drives home the point that such things must not be used except in rare extraordinary cases. If unwrap() were named UNWRAP_OR_PANIC(), it would be used much less glibly. Even more, I wish ther…
right and if the language designers named it UNWRAP_OR_PANIC() then people would rightfully be asking why on earth we can't just use a try-catch around code and have an easier life
...and you can? try-catch is usually less ergonomic than the various ways you can inspect a Result.
try {
data = some_sketchy_function();
} catch (e) {
handle the error;
}
result = some_sketchy_function();
if let Err(e) = result {
handle the error;
}
data = some_sketchy_function();
data = some_sketchy_function().UNWRAP_OR_PANIC();
What I suspect you mean, because it's a better argument, is:
try {
sketchy_function1();
sketchy_function2();
sketchy_function3();
sketchy_function4();
} catch (e) {
...
}