Speaking of resiliency, the entire Bot Management module doesn't seems to be a critical part of the system, so for example, what happens if that module goes down for an hour? the other parts of the system should work. So I would rank every module and it's role in the system, and would design it in a way that when a non-critical module fails, other parts still can function.
Cloudflare outage on November 18, 2025 post mortem
621–630 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#622Earlier quoted context omitted.
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…
They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…
> Why did they think this was a cyberattack
Isn’t getting cyberattacked their core business?
Re: Cloudflare outage on November 18, 2025 post mortem
#623This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
They should link this article in the docs for `unwrap()`.
Re: Cloudflare outage on November 18, 2025 post mortem
#624This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
Handling the error still would've returned a 5xx in this case, since the config file was still over the limit of features the service could handle.
Re: Cloudflare outage on November 18, 2025 post mortem
#625Seems like a substantial fraction of the web was brought down because of a coding error that should have been caught in CI by a linter.
These folks weren't operating for charity. They were highly paid so-called professionals.
Who will be held accountable for this?
Re: Cloudflare outage on November 18, 2025 post mortem
#626Earlier quoted context omitted.
I don't understand why they didn't validate and sanitize the new config file revision. If bad(whatever that reason is) throw an error and revert back to previous version. You don't need to take down the whole internet for that.
Same as for almost every bug I think: the dev in question hadn't considered that the input could be bad in the way that it turned out to be. Maybe they were new, or maybe they hadn't slept much because of a newborn baby, or maybe they thought it was a reasonable assumption that there would never be more than 200 ML features in the array in question. I don't think this developer will ever make the same mistake again a…
> Maybe they were new, or maybe they hadn't slept much because of a newborn baby
Reminds me of House of Dynamite, the movie about nuclear apocalypse that really revolves around these very human factors. This outage is a perfect example of why relying on anything humans have built is risky, which includes the entire nuclear apparatus. “I don’t understand why X wasn’t built in such a way that wouldn’t mean we live in an underground bunker now” is the sentence that comes to mind.
Re: Cloudflare outage on November 18, 2025 post mortem
#627Earlier quoted context omitted.
I'm not a fan of rust, but I don't think that is the only takeaway. All systems have assumptions about their input and if the assumption is violated, it has to be caught somewhere. It seems like it was caught too deep in the system. Maybe the validation code should've handled the larger size, but also the db query produced something invalid. That shouldn't have ever happened in the first place.
The takeaway here isn’t about Rust itself, but that the Rust marketing crew’s claims that we constantly read on HN and elsewhere about the Result type magically saving you from making mistakes is not a good message to send.
They would also tell you that .unwrap() has no place in production code, and should receive as much scrutiny as an `unsafe` block in code review :)
The point of option is the crash path is more verbose and explicit than the crash-free path. It takes more code to check for NULL in C or nil in Go; it takes more code in Rust to not check for Err.
Re: Cloudflare outage on November 18, 2025 post mortem
#628Earlier quoted context omitted.
This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a n…
Exactly! Sometimes exploding is simply the least bad option, and is an entirely sensible approach.
In this case it definitely wasn’t the least bad option though.
Re: Cloudflare outage on November 18, 2025 post mortem
#629This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
> This is the multi-million dollar .unwrap() story. While there are certainly many things to admire about Rust, this is why I prefer Golang's "noisy" error handling. In golang that would be either: feature_values, err := features.append_with_names(...) And the compiler would have complained that this value of `err` was unused; or you'd write: feature_values, _ := features.append_with_names(...) And it would be far mo…
That may be me, but `.unwrap()` is much more obvious than `_`:
- it's literally written out that you're assuming it to be Ok
- there are no indications that the `_` is an error: it could very well be some other return value from the function. in your example, it could be the number of appended features, etc
That's why Go's error handling is indeed noisy: it's noise and you reduce noise by not handling errors. Rust's is terse yet verbose: if you add stuff it's because you're doing something wrong. You explicitly spelled out the error is being ignored.
Re: Cloudflare outage on November 18, 2025 post mortem
#630Earlier quoted context omitted.
Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code. And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.
[lints.clippy] dbg_macro = "deny" unwrap_used = "deny" expect_used = "deny"
Exactly. This should be the default for production code at companies like Cloudflare.