Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

471–480 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#471

Post by sznio »

jcalvinowens wrote: post #275
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

>If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.

Re: Cloudflare outage on November 18, 2025 post mortem

#472

Post by branko_d »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Safe things should be easy, dangerous things should be hard.

This .unwrap() sounds too easy for what it does, certainly much easier than having an entire try..catch block with an explicit panic. Full disclosure: I don't actually know Rust.

Re: Cloudflare outage on November 18, 2025 post mortem

#473

Post by pas »

abalone wrote: post #380
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Rolling out new code should be done differently than rolling out new data to fight bots.

If every time there's a new bot someone needs to write code that can blow up their whole service, maybe they need to iterate a bit on this design?

Re: Cloudflare outage on November 18, 2025 post mortem

#474

Post by brianpan »

smj-edison wrote: post #207
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Isn't the point of this article that pieces of infrastructure don't go down to root causes, but due to bad combinations of components that are correct individually? After reading "engineering a safer world", I find root cause analysis rather reductionistic, because it wasn't just an unwrap, it was that the payload was larger than normal, because of a query that didn't select by database, because a clickhouse made mor…

You're right. A good postmortem/root cause analysis would START from "unwrap" and continue from there.

You might start with a basic timeline of what happened, then you'd start exploring: why did this change affect so many customers (this would be a line of questioning to find a potential root cause), why did it take so long to discover or recover (this might be multiple lines of questioning), etc.

Re: Cloudflare outage on November 18, 2025 post mortem

#475

Post by ergocoder »

richardwhiuk wrote: post #470
karel-3d wrote: post #468

Earlier quoted context omitted.

As a gopher I never understand why is there so many unwraps in an average rust code. Average Go code has much less panics than Rust has unwraps, which are functionally equivalent.

Because Go silently gives you zero/null instead

which mean an unexpected behavior could go unnoticed for a long time.

I'd prefer a loud crash over that.

Re: Cloudflare outage on November 18, 2025 post mortem

#476

Post by dwattttt »

shadowgovt wrote: post #400
helloericsf wrote: post #256

Earlier quoted context omitted.

Not a DBA, how do you do DB permission rollout gating?

It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems. The file should be versioned and rollout of new versions should be staged. (There is definitely a trade-off; often times in the security critical path, you want to go as fast as possible because changes may be blocking a malicious actor. But if yo…

> It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.

That feature file is generated every 5 minutes at all times; the change to permissions was rolled out gradually over the clickhouse cluster, and whether a bad version of that file was generated depended on whether the part of the cluster that had the bad permissions generated the file.

Re: Cloudflare outage on November 18, 2025 post mortem

#477

Post by speedgoose »

bigstrat2003 wrote: post #347
echelon wrote: post #327

Earlier quoted context omitted.

Panics should be explicit, not implicit. unwrap() should effectively work as a Result where the user must manually invoke a panic in the failure branch. Make special syntax if a match and panic is too much boilerplate. This is like an implicit null pointer exception that cannot be statically guarded against. I want a way to statically block any crates doing this from my dependency chain.

unwrap is explicit.

Not explicit enough, apparently.

Re: Cloudflare outage on November 18, 2025 post mortem

#478

Post by ergocoder »

fredoralive wrote: post #467
jcalvinowens wrote: post #317

Earlier quoted context omitted.

> Instead, the routine likely should have worked within `Result`. But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same. Maybe you're saying "it's better to be explicit", as a broad generalization I don't disagree with that. But that has nothing to do with the actual bug here, which was that the invariant failed. How they choose to implement checking and failing the invar…

A failed config load probably shouldn't be a fatal error if a valid config is already loaded?

Hard to say. Why would you load a new config if a valid config is already loaded?

Maybe the new config has a new update. Who knows? Do we want to keep operating on the old config? Maybe maybe not.

But operating on old config when you don't want to is definitely worse.

Re: Cloudflare outage on November 18, 2025 post mortem

#479

Post by speedgoose »

karel-3d wrote: post #468
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

As a gopher I never understand why is there so many unwraps in an average rust code. Average Go code has much less panics than Rust has unwraps, which are functionally equivalent.

The average golang code segfaults by design.
Post reply on HN