Any feature failing should still allow the traffic to continue. This should be the first bullet in the future actions list.
Cloudflare outage on November 18, 2025 post mortem
631–640 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#632This incident feels like a strong argument for stricter guardrails around internal config propagation
Re: Cloudflare outage on November 18, 2025 post mortem
#633Earlier quoted context omitted.
> This is the multi-million dollar .unwrap() story. While there are certainly many things to admire about Rust, this is why I prefer Golang's "noisy" error handling. In golang that would be either: feature_values, err := features.append_with_names(...) And the compiler would have complained that this value of `err` was unused; or you'd write: feature_values, _ := features.append_with_names(...) And it would be far mo…
But I could screw it up in Go, if I made the same assumptions fvs, err := features.AppendWithNames(..) if err != nil { // this will NEVER break panic(err) } Ultimately I don't think language design can be the sole line of defence against system failures; it can only guide developers to think about error cases
Right, but the point isn't to make errors impossible; the point is to have them be 1) less likely to write, and 2) easier to spot on review.
People's biggest complaints about golang's errors:
1. You have to _TYPE_OUT_ what to do on EVERY.SINGLE.ERROR. SOO BOORING!
2. They clutter up the code and make it look ugly.
Rust is so much cleaner and more convenient (they say)! Just add ?, or .unwrap()!
Well, with ".unwrap()", you can type it fast enough that you're on to the next problem before it occurs to your brain to think about what to do if there is an error. Whereas, in golang, by the time you type in, "if err != nil {", you've broken the flow enough that now you're much more likely to be thinking, "Hmm, could this ever fail? What should we do if it does?" That break in flow is annoying, but necessary.
And ".unwrap()" looks so unassuming, it's easy to overlook on review; that "panic()" looks a lot more dangerous, and again, would be more likely to trigger a reviewer into thinking, "Wait, is it OK if this thing panics? Is this really so unlikely to happen?"
Renaming it `.unwrap_or_panic()` would probably help with both.
Re: Cloudflare outage on November 18, 2025 post mortem
#634We shouldn't be having critical internet-wide outages on a monthly basis. Something is systematically wrong with the way we're architecting our systems.
Cloudflare, Azure, and other single points of failure are solving issues inherent to webhosting, and those problems have become incredibly hard due to the massive scale of bad actors and the massive complexity of managing hardware and software. What would you propose to fix it? The fixed cost of being DDoS-proof is in the hundreds of millions of dollars.
Saying "other single points of failure" makes no sense whatsoever.
Re: Cloudflare outage on November 18, 2025 post mortem
#635Earlier quoted context omitted.
I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…
> I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate.
I'm so jealous. I've written postmortems for major incidents at a previous job: a few hours to write, a week of bikeshedding by marketing and communication and tech writers and ... over any single detail in my writing. Sanitizing (hide a part), simplifying (our customers are too dumb to understand), etc, so that the final writing was "true" in the sense that it "was not false", but definitely not what I would call "true and accurate" as an engineer.
Re: Cloudflare outage on November 18, 2025 post mortem
#636Earlier quoted context omitted.
Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…
> at least do `expect("damn it, how did this happen")` That gives you the same behavior as unwrap with a less useful error message though. In theory you can write useful messages, but in practice (and your example) expect is rarely better than unwrap in modern rust
We shouldn't be using unwrap() or expect() at all.
This is Rust's Null Pointer Exception.
unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features.
The language should grow the ability to mark this code as dangerous, and we should have static tools to exclude this code from our dependency tree.
I don't want some library I use to `unwrap()` and cause my application to crash because I didn't anticipate their stupid panic.
Rust developers have clearly leaned on this crutch far too often:
https://github.com/search?q=unwrap%28%29+language%3ARust&typ...
The Rust team needs to plug this leak.
Re: Cloudflare outage on November 18, 2025 post mortem
#637This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
Feels like a case where safety guarantees of Rust lulled them into thinking the edge cases were covered
Re: Cloudflare outage on November 18, 2025 post mortem
#638Earlier quoted context omitted.
blaming the language is not the way to approach this. if an engineer writes bad code that’s the engineers fault, not the languages. this was bad code that should have never hit production, it is not a rust language issue.
No. Don't say "you're holding it wrong". The language says "safe" on the tin. It advertises safety. This shouldn't be possible. This is a null pointer. In Rust. Unwrap needs to die. We should all fight to remove it.
you either misunderstand the rust ethos or are intentionally misrepresenting it.
safe refers to memory safety.
once again, if you write bad code, that’s your fault, not the languages. this is a feature of rust that was used incorrectly.
Re: Cloudflare outage on November 18, 2025 post mortem
#639Earlier quoted context omitted.
See the immediately preceding sentence. I'm fine with allocation failures. I don't want stupid unwrap()s, improper slice access, or other stupid and totally preventable behavior. There are things inside the engineer's control. I want that to not panic.
Your pair of posts is very interesting to me. Can you share with me: What is your programming environment such that you are "fine with allocation failures"? I'm not doubting you, but for me, if I am doing systems programming with C or C++, my program is doomed if a malloc fails! When I saw your post, I immediately thought: Am I doing it wrong? If I get a NULL back from malloc(), I just terminate with an error message…
If I can't allocate memory, I'm typically okay with the program terminating.
I don't want dependencies deciding to unwrap() or expect() some bullshit and that causing my entire program to crash because I didn't anticipate or handle the panic.
Code should be written, to the largest extent possible, to mitigate errors using Result. This is just laziness.
I want checks in the language to safeguard against lazy Rust developers. I don't want their code in my dependency tree, and I want static guarantees against this.
edit: I just searched unwrap() usage on Github, and I'm now kind of worried/angry:
https://github.com/search?q=unwrap%28%29+language%3ARust&typ...
A lot of this is just pure laziness.
Re: Cloudflare outage on November 18, 2025 post mortem
#640This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
Interesting to see Rust error handling flunk out in practice. It may be that forcing handling at every call tends to makes code verbose, and devs insensitized to bad practice. And the diagnostic Rust provided seems pretty garbage. There is bad practice here too -- config failure manifesting as request failure, lack of failing to safe, unsafe rollout, lack of observability. Back to language design & error handling. My…
I don't understand how your takeaway is that this is a language flaw other than to assume that you have some underlying disdain for Rust. That's fine, but state it clearly please.
The end result would've been the exact same if they "handled" the error: a bunch of 500s. The language being used doesn't matter if an invariant in your system is broken.