How many changes to production systems does Cloudflare make throughout a day? Are they a part of any change management process? That would be the first place I would check after a random outage, recent changes.
Cloudflare outage on November 18, 2025 post mortem
671–680 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#672Earlier quoted context omitted.
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…
Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code. And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.
What would the difference be if they had enforced no unwraps/expects/slicing and instead logged the error and returned a 500?
As the user, I can't tell the difference, but it might have sped up their recovery a bit.
Re: Cloudflare outage on November 18, 2025 post mortem
#673Earlier quoted context omitted.
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
Why was Warp in London disabled temporarily. No mention of that change was discussed in the RCA despite it being called out in an update. For London customers this made the impact more severe temporarily.
We incorrectly thought at the time it was attack traffic coming in via WARP into LHR. In reality it was just that the failures started showing up there first because of how the bad file propagated and where it was working hours in the world.
Re: Cloudflare outage on November 18, 2025 post mortem
#674I'm honestly curious what culturally is going on inside Cloudflare given they've had a few outages this year.
Re: Cloudflare outage on November 18, 2025 post mortem
#675This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…
I would say that whilst this is a good top down view, that `.unwrap()` should have been caught at code-review and not allowed. Clippy rule could have saved a lot of money.
That and why the hell wasn't their alerting showing up colossal amount of panics in their bot manager thing?
Re: Cloudflare outage on November 18, 2025 post mortem
#676> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was tri…
> much better than their completely false “checking the security of your connection” message The exact wording (which I can easily find, because a good chunk of the internet gives it to me, because I’m on Indian broadband): > example.com needs to review the security of your connection before proceeding. It bothers me how this bald-faced lie of a wording has persisted. (The “Verify you are human by completing the acti…
Next time open your dev console in your window and look at how much is going on in the background.
Re: Cloudflare outage on November 18, 2025 post mortem
#677Earlier quoted context omitted.
I don't think these are realistic requirements for any engineered system to be honest. Realistic is to have contingencies for such cases, which are simply errors. But the case for Cloudflare here is complicated. Every engineer is very free to make a better system though.
What is not realistic? To do simple input validation on data that has the potential to break 20% of the internet? To not have a system in place to rollback to the latest known state when things crash? Cloudflare builds a global scale system, not an iphone app. Please act like it.
> To do simple input validation on data that has the potential to break 20% of the internet?
There will always be bugs in code, even simple code, and sometimes those things don't get caught before they cause significant trouble.
The failing here was not having a quick rollback option, or having it and not hitting the button soon enough (even if they thought the problem was probably something else, I think my paranoia about my own code quality is such that I would have been rolling back much sooner just in case I was wrong about the “something else”).
Re: Cloudflare outage on November 18, 2025 post mortem
#678Earlier quoted context omitted.
Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen a…
Fair points; there's certainly a balance to be struck between raising awareness and amplifying, and I admittedly have no idea where that line is.
I'm not the person you were replying to, but there is a rule I often see about not directly replying/quote tweeting because "engagement" appears to boost support for the ideas expressed. The recommendation then, would be to screenshot it (often with the username removed) and link to that.
Re: Cloudflare outage on November 18, 2025 post mortem
#679> That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. > The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That…
Yep, a decent canary mechanism should have caught this. There's a trade off between canarying and rollout speed, though. If this was a system for fighting bots, I'd expect it to be optimized for the latter.
Presumably optimal rollout speed entails something like or as close to ”push it everywhere all at once and activate immediately” that you can get — that’s fine if you want to risk short downtime rather than delays in rollout, what I don’t understand is why the nodes don’t have any independent verification and rollback mechanism. I might be underestimating the complexity but it really doesn’t sound much more involved than a process launching another process, concluding that it crashed and restarting it with different parameters.
Re: Cloudflare outage on November 18, 2025 post mortem
#680Earlier quoted context omitted.
Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code. And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.
What would the difference be if they had enforced no unwraps/expects/slicing and instead logged the error and returned a 500? As the user, I can't tell the difference, but it might have sped up their recovery a bit.
You could argue that explicitly writing down the assumption would make it clearer to yourself and your reviewer that it might be overly optimistic.