Matt, Looking forward in regaining Elon's and his team trust to use CF again.
Cloudflare outage on November 18, 2025 post mortem
91–100 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#92"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our stat…
Quite possibly it was due to high traffic.
IDK Atlassian Statuspage clientele, but it's possible Cloudflare is much larger than usual.
Re: Cloudflare outage on November 18, 2025 post mortem
#93Earlier quoted context omitted.
Probably because this case was something more akin to an assert than an error check.
Fly writes a lot of Rust, do you allow `unwrap()` in your production environment? At Modal we only allow `expect("...")` and the message should follow the recommended message style[1]. I'm pretty surprised that Cloudflare let an unwrap into prod that caused their worst outage in 6 years. 1. https://doc.rust-lang.org/std/option/enum.Option.html#recomm...
After The Great If-Let Outage Of 2024, we audited all our code for that if-let/rwlock problem, changed a bunch of code, and immediately added a watchdog for deadlocks. The audit had ~no payoff; the watchdog very definitely did.
I don't know enough about Cloudflare's situation to confidently recommend anything (and I certainly don't know enough to dunk on them, unlike the many Rust experts of this thread) but if I was in their shoes, I'd be a lot less interested in eradicating `unwrap` everywhere and more in making sure than an errant `unwrap` wouldn't produce stable failure modes.
But like, the `unwrap` thing is all programmers here have to latch on to, and there's a psychological self-soothing instinct we all have to seize onto some root cause with a clear fix (or, better yet for dopaminergia, an opportunity to dunk).
A thing I really feel in threads like this is that I'd instinctively have avoided including the detail about an `unwrap` call --- I'd have worded that part more ambiguously --- knowing (because I have a pathological affinity for this community) that this is exactly how HN would react. Maybe ironically, Prince's writing is a little better for not having dodged that bullet.
Re: Cloudflare outage on November 18, 2025 post mortem
#94"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our stat…
We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.
The status page is hosted on AWS Cloudfront, right? It sure looks like Cloudfront was overwhelmed by the traffic spike, which is a bit concerning. Hope we'll see a post from their side.
Re: Cloudflare outage on November 18, 2025 post mortem
#95As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.
I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.
Re: Cloudflare outage on November 18, 2025 post mortem
#96This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language
Re: Cloudflare outage on November 18, 2025 post mortem
#97> work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditio…
Global configuration is useful for low response times to attacks, but you need to have very good ways to know when a global config push is bad and to be able to rollback quickly.
In this case, the older proxy's "fail-closed" categorization of bot activity was obviously better than the "fail-crash", but every global change needs to be carefully validated to have good characteristics here.
Having a mapping of which services are downstream of which other service configs and versions would make detecting global incidents much easier too, by making the causative threads of changes more apparent to the investigators.
Re: Cloudflare outage on November 18, 2025 post mortem
#98Earlier quoted context omitted.
It seems they had this continous rollout for the config service, but the services consuming this were affected even by small percentage of these config providers being faulty, since they were auto updating every few minutes their configs. And it seems there is a reason for these updating so fast, presumably having to react to threat actors quickly.
It's in everyone's interest to mitigate threats as quickly as possible. But it's of even greater interest that a core global network infrastructure service provider not DOS a significant proportion of the Internet by propagating a bad configuration too quickly. The key here is to balance responsiveness against safety, and I'm not sure they struck the right balance here. I'm just glad that the impact wasn't as long an…
This isn't really "configuration" so much as it is "durable state" within the context of this system.
Re: Cloudflare outage on November 18, 2025 post mortem
#99> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually…
What people are saying is that idiomatic prod rust doesn't use unwrap/expect (both of which panic on the "exceptional" arm of the value) --- instead you "match" on the value and kick the can up a layer on the call chain.
What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than bot mitigation you'd still have the same symptom of getting 500'd.
Re: Cloudflare outage on November 18, 2025 post mortem
#100So, to recap: - Their database permissions changed unexpectedly (??) - This caused a 'feature file' to be changed in an unusual way (?!) - Their SQL query made assumptions about the database; their permissions change thus resulted in queries getting additional results, permitted by the query - Changes were propagated to production servers which then crashed those servers (meaning they weren't tested correctly) - They…
Looks like you have the perfect window to disrupt them with a superior product.