Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

391–400 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#391

Post by hoppp »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

You write so much rust you causally apply unwrap now to everything?

Rust compiler is a god of sorts, or at least a law of nature haha

Way to comment and go instantly off topic

Re: Cloudflare outage on November 18, 2025 post mortem

#392

Post by smt88 »

jdlyga wrote: post #316

We shouldn't be having critical internet-wide outages on a monthly basis. Something is systematically wrong with the way we're architecting our systems.

Cloudflare, Azure, and other single points of failure are solving issues inherent to webhosting, and those problems have become incredibly hard due to the massive scale of bad actors and the massive complexity of managing hardware and software.

What would you propose to fix it? The fixed cost of being DDoS-proof is in the hundreds of millions of dollars.

Re: Cloudflare outage on November 18, 2025 post mortem

#393

Post by perching_aix »

tptacek wrote: post #93
thundergolfer wrote: post #78

Earlier quoted context omitted.

Fly writes a lot of Rust, do you allow `unwrap()` in your production environment? At Modal we only allow `expect("...")` and the message should follow the recommended message style[1]. I'm pretty surprised that Cloudflare let an unwrap into prod that caused their worst outage in 6 years. 1. https://doc.rust-lang.org/std/option/enum.Option.html#recomm...

After The Great If-Let Outage Of 2024, we audited all our code for that if-let/rwlock problem, changed a bunch of code, and immediately added a watchdog for deadlocks. The audit had ~no payoff; the watchdog very definitely did. I don't know enough about Cloudflare's situation to confidently recommend anything (and I certainly don't know enough to dunk on them, unlike the many Rust experts of this thread) but if I was…

Sounds like if nothing else, additional attention around (their?) use of unwrap() is still warranted from where you're sitting then though, no? I don't think there's anything wrong with flagging that.

It's one thing to not want to be the one to armchair it, but that doesn't mean that one has to suppress their normal and obvious reactions. You're allowed to think things even if they're kitsch, you too are human, and what's kitsch depends and changes. Applies to everyone else here by extension too.

Re: Cloudflare outage on November 18, 2025 post mortem

#394

Post by AtNightWeCode »

rafaelmn wrote: post #362
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

That's such a bad take after reading the article. If you're going to write a system that preallocates and is based on hard assumptions about max size - the panic/unwrap approach is reasonable. The config bug reaching prod without this being caught and pinpointed immediately is the strange part.

Exactly. The newbie mistake in SQL is also way worse than this. But the whole design is also bad. Clearly implementing things at the wrong place.

And, it took like over an hour between the problem started til my sites went down. That is just crazy.

Re: Cloudflare outage on November 18, 2025 post mortem

#395

Post by thatoneengineer »

rafaelmn wrote: post #362
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

That's such a bad take after reading the article. If you're going to write a system that preallocates and is based on hard assumptions about max size - the panic/unwrap approach is reasonable. The config bug reaching prod without this being caught and pinpointed immediately is the strange part.

I agree there's no way to soft-error this, though "truncate and raise an alert" is arguably the better pattern.

Re: Cloudflare outage on November 18, 2025 post mortem

#400

Post by shadowgovt »

helloericsf wrote: post #256
shadowgovt wrote: post #195

Earlier quoted context omitted.

In addition, it looks like this system wasn't on any kind of 1%/10%/50%/100% rollout gating. Such a rollout would trivially have shown the poison input killing tasks.

Not a DBA, how do you do DB permission rollout gating?

It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.

The file should be versioned and rollout of new versions should be staged.

(There is definitely a trade-off; often times in the security critical path, you want to go as fast as possible because changes may be blocking a malicious actor. But if you move too fast, you break things. Here, they had a potential poison input in the pathway for synchronizing this state and Murphy's Law suggests it was going to break eventually, so the question becomes "How much damage can we tolerate when it does?")

Post reply on HN