Cloudflare outage on November 18, 2025 post mortem

That’s an invariant meant to be expressed by your type system — and it is.

You’ve failed to model your invariants in your API — and thus the type system — if you ever reach a point where an engineer has to manually assess and assert whether “cannot” applies.

You will get it wrong. That is bad code.

The oposing views here are not "hope and prayers" vs "good engineering", it's assuming things will fail at every stage vs assuming one can build a layer of abstraction that is flawless, on top of which we can build.

Resilient systems trump "correct" systems, and I would pick a system designed under the assumption that fake errors will be injected regularly, that process will be killed at random, that entire rack of machines will be unplugged at random at any time, that whole datacenters will be put off grid for fun, over a system that's been "proven correct", any day. I though it was common knowledge.

Of coursre I'm not arguing against proving that a software is correct. I would actually argue that some formal methods would come handy to model these kind of systemic failures and reveal the worste cases with largest blast radius.

But considering the case at hand, the code for that FL2 bot had an assertion regarding the size of received data and that was a valid assertion, and the process decided to panic, and that was the right decision. What was not right was the lack of instrumentation that should have made these failures obvious, and the fact that the user queries failed when that non-essential bot failed, instead of bypassing that bot.

What he seems to be saying is that "obviously in C I would be checking the icon handle for being non-null so clearly error value handling is superior" but this is only obvious to someone knowing the API and checking values for validity has to be done in exception based code too. It's just that exception based code doesn't pretend that it cannot panic somewhere where you don't know. The default, better assumption for programming is that you don't know what this code is doing but it should just work. Unchecked exception handling is the best way to fit that paradigm, you should not have to care about every single line and what it does and constantly sort of almost obsessively check error values of all the APIs you ever use to have this false hope that it cannot panic because you did your duty. No, it can still panic and all this error checking is not helping you program better or more clearly or faster. It swamps the code with so many extra lines that it's practically double the size. All this makes it less clear and that is also what his post shows.

Cloudflare's incident report is written clearly and explicitly, so based on my own understanding, I’m going to try reproducing this outage. Already completed:

CK cluster Permission change triggering data doubling Cache propagation Unaffected proxy services Proxy services with bot score errors

TODO:

unwrap panic during pre-allocation of cache Full demonstration of the entire outage process

https://github.com/Laotree/reproduce_cf20251118

In practice, programmers don't find it easy to keep in mind that certain functions might throw. This is a real problem with unchecked exceptions and with C-style error codes that sloppy programmers might ignore entirely.

> [...] on which properties to set first can happen in any language

A carefully designed library using a statically typed functional language, especially a pure functional language, might sometimes be able to eliminate such hidden ordering bugs.

Rust used to have a feature to help the compiler detect invalid ordering of imperative operations, called typestates. This feature has since been mostly removed, though, as it saw little use. [0]

> isn't related at all to whether you use try-catch handling or error values as return codes

I guess Chen is assuming a reasonably diligent programmer who makes a habit of never discarding status/error values returned by functions. C++'s [[nodiscard]] can help ensure this.

(Of course, outside of C++, those aren't the only options. Idiomatic Haskell and Zig code forces the programmer to explicitly handle the possibility of an error. Same goes for Java's checked exceptions.)

> What he seems to be saying is that "obviously in C I would be checking the icon handle for being non-null so clearly error value handling is superior"

I don't think he's exactly arguing for the C-style approach, he's more just criticizing exceptions, especially unchecked exceptions. I agree the C-style approach has considerable problems.

> It's just that exception based code doesn't pretend that it cannot panic somewhere where you don't know.

With checked exceptions, you know precisely which operations can throw.

> Unchecked exception handling is the best way to fit that paradigm, you should not have to care about every single line and what it does and constantly sort of almost obsessively check error values of all the APIs you ever use to have this false hope that it cannot panic because you did your duty

You do need to care about every line, or your plausible-looking code is likely to misbehave when an exception occurs, as Chen's post demonstrates. Unchecked exceptions deprive the compiler of the ability to ensure good exception-handling coverage. There is no error-handling model that allows to programmer to write good code by pretending errors won't arise.

(I presume that by panic you mean throw an unchecked exception.)

[0] https://cliffle.com/blog/rust-typestate/