Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

311–320 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#311

Post by echelon »

speed_spread wrote: post #264
wrs wrote: post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

Pet peeve: unwrap() should be deprecated and renamed or_panic(). More consistent with the rest of stdlib methods and appropriately scarier.

A lot of stuff should be done about the awful unwrap family of methods.

A few ideas:

- It should not compile in production Rust code

- It should only be usable within unsafe blocks

- It should require explicit "safe" annotation from the engineer. Though this is subject to drift and become erroneous.

- It should be possible to ban the use of unsafe in dependencies and transitive dependencies within Cargo.

Re: Cloudflare outage on November 18, 2025 post mortem

#312

Post by reassess_blind »

bri3d wrote: post #260

Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random…

Once they figured it out they didn't have a way to load in a new feature file, had to figure that out, and then restart every machine.

Re: Cloudflare outage on November 18, 2025 post mortem

#313

Post by hedora »

__turbobrew__ wrote: post #130
olivia-banks wrote: post #99

Earlier quoted context omitted.

What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than…

Presumably you kick up the error to a level that says “if parsing new config fails, keep the old config”

The config file subsystem was where the bug lived, not the code with the unwrap, so this sort of change is a special case of “make the unwrap never fail and then fix the API so it is not needed”.

Re: Cloudflare outage on November 18, 2025 post mortem

#314

Post by ynx0 »

eastdakota wrote: post #95
yen223 wrote: post #57

Earlier quoted context omitted.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…

How do you guys handle redaction? I'm sure even when trusted individuals are in charge of authoring, there's still a potential of accidental leakage which would probably be best mitigated by a team specifically looking for any slip ups.

Thanks for the insight.

Re: Cloudflare outage on November 18, 2025 post mortem

#315

Post by bigstrat2003 »

Terr_ wrote: post #276
dehrmann wrote: post #231

Earlier quoted context omitted.

It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.

"Checked Exceptions Are Actually Good" gang, rise up! :p I think adoption would have played out very different if there had only been some more syntactic-sugar. For example, an easy syntax for saying: "In this method, any (checked) DeepException e that bubbles up should immediately be replaced by a new (checked) MylayerException(e) that contains the original one as a cause. We might still get lazy programmers making…

I'm with you! Checked exceptions are actually good and the hate for them is super short sighted. The exact same criticisms levied at checked exceptions apply to static typing in general, but people acknowledge the great value static types have for preventing errors at compile time. Checked exceptions have that same value, but are dunked on for some reason.

Re: Cloudflare outage on November 18, 2025 post mortem

#317

Post by jcalvinowens »

AdieuToLogic wrote: post #309
jcalvinowens wrote: post #275

Earlier quoted context omitted.

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

>> This is the multi-million dollar .unwrap() story. > That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same. Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assu…

> Instead, the routine likely should have worked within `Result`.

But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same.

Maybe you're saying "it's better to be explicit", as a broad generalization I don't disagree with that.

But that has nothing to do with the actual bug here, which was that the invariant failed. How they choose to implement checking and failing the invariant in the semantics of the chosen language is irrelevant.

Re: Cloudflare outage on November 18, 2025 post mortem

#318

Post by aw1621107 »

echelon wrote: post #303
dap wrote: post #301

Earlier quoted context omitted.

I'm not sure if this is serious or not, but to take it at face value: the value of this sort of thing in Rust is not that it prevents crashes altogether but rather that it prevents _implicit_ failures. It forces a programmer to make the explicit choice of whether to crash. There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this)…

The language semantics do not surface `unwrap` usage or make any guarantees. It should be limited to use in `unsafe` blocks. > There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe. I would prefer the boiler plate of a match / if-else / if let, etc. to call atte…

> It should be limited to use in `unsafe` blocks.

That would be a fairly significant expansion of what `unsafe` means in Rust, to put it lightly. Not to mention that I think doing so would not really accomplish anything; marking unwrap() `unsafe` would not "surface `unwrap` usage" or "make any guarantees", as it's perfectly fine for safe functions to contain `unsafe` blocks with zero indication of such in the function signature and.

Re: Cloudflare outage on November 18, 2025 post mortem

#319

Post by _zagj »

wrs wrote: post #86
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

> It seems people have a blind spot for unwrap

Not unlike people having a blind spot for Rust in general, no?

Re: Cloudflare outage on November 18, 2025 post mortem

#320

Post by echelon »

aw1621107 wrote: post #318
echelon wrote: post #303

Earlier quoted context omitted.

The language semantics do not surface `unwrap` usage or make any guarantees. It should be limited to use in `unsafe` blocks. > There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe. I would prefer the boiler plate of a match / if-else / if let, etc. to call atte…

> It should be limited to use in `unsafe` blocks. That would be a fairly significant expansion of what `unsafe` means in Rust, to put it lightly. Not to mention that I think doing so would not really accomplish anything; marking unwrap() `unsafe` would not "surface `unwrap` usage" or "make any guarantees", as it's perfectly fine for safe functions to contain `unsafe` blocks with zero indication of such in the functio…

> fairly significant expansion of what `unsafe` means in Rust

I want an expansion of panic free behavior. We'll never get all the way there due to allocations etc., but this is the class of error the language is intended to fix.

This turned into a null pointer, which is exactly what Rust is supposed to quench.

I'll go as far as saying I would like to statically guarantee none of my dependencies use the unwrap() methods. We should be able to design libraries that provably avoid panics to the greatest extent possible.

Unwrap is an easy loss on a technicality.

Post reply on HN