Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

301–310 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#301

Post by dap »

echelon wrote: post #288
jcalvinowens wrote: post #275

Earlier quoted context omitted.

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

[flagged]

I'm not sure if this is serious or not, but to take it at face value: the value of this sort of thing in Rust is not that it prevents crashes altogether but rather that it prevents _implicit_ failures. It forces a programmer to make the explicit choice of whether to crash.

There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe.

Re: Cloudflare outage on November 18, 2025 post mortem

#302

Post by antonvs »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is textbook "parse, don't validate" anti-pattern.

How so? “Parse, don’t validate” implies converting input into typed values that prevent representation of invalid state. But the parsing still needs to be done correctly. An unchecked unwrap really has nothing to do with this.

Re: Cloudflare outage on November 18, 2025 post mortem

#303

Post by echelon »

dap wrote: post #301
echelon wrote: post #288

Earlier quoted context omitted.

[flagged]

I'm not sure if this is serious or not, but to take it at face value: the value of this sort of thing in Rust is not that it prevents crashes altogether but rather that it prevents _implicit_ failures. It forces a programmer to make the explicit choice of whether to crash. There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this)…

The language semantics do not surface `unwrap` usage or make any guarantees. It should be limited to use in `unsafe` blocks.

> There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe.

I would prefer the boiler plate of a match / if-else / if let, etc. to call attention to it. If you absolutely must explicitly panic. Or better - just return an error Result.

It doesn't matter how smart your engineers are. A bad unwrap can sneak in through refactors, changing business logic, changing preconditions, new data, etc.

Re: Cloudflare outage on November 18, 2025 post mortem

#304

Post by [deleted] »

chrisweekly wrote: post #292
jcalvinowens wrote: post #275

Earlier quoted context omitted.

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

semantic? or pedantic?

[deleted]

Re: Cloudflare outage on November 18, 2025 post mortem

#305

Post by SilverElfin »

mh- wrote: post #285
SilverElfin wrote: post #268

Earlier quoted context omitted.

[flagged]

Why give this sort of content more visibility/reach? I'm sure that's not your intent, so I hope my comment gives you an opportunity to reflect on the effects of syndicating such stupidity, no matter what platform it comes from.

Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen as a nation to grant for our own benefit.

The legislation that MTG (Marjorie Taylor Green) just proposed a few days ago to ban H1B entirely, and the calls to ban other visa types, is going to have a big negative impact on the tech industry and American innovation in general. The social media stupidity is online but it gives momentum to the actual real life legislation and other actions the administration might take. Many congress people are seeing the online sentiment and changing their positions in response, unfortunately.

Re: Cloudflare outage on November 18, 2025 post mortem

#306

Post by hedora »

evil-olive wrote: post #168
olivia-banks wrote: post #99

Earlier quoted context omitted.

What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than…

> What happens to it up the callstack? as they say in the post, these files get generated every 5 minutes and rolled out across their fleet. so in this case, the thing farther up the callstack is a "watch for updated files and ingest them" component. that component, when it receives the error, can simply continue using the existing file it loaded 5 minutes earlier. and then it can increment a Prometheus metric (or si…

Given that the bug was elsewhere in the system (the config file parser spuriously failed), it’s hard to justify much of what you suggested.

Panics should be logged, and probably grouped by stack trace for things like prometheus (outside of process). That handles all sorts of panic scenarios, including kernel bugs and hardware errors, which are common at cloudflare scale.

Similarly, mitigating by having rapid restart with backoff outside the process covers far more failure scenarios with far less complexity.

One important scenario your approach misses is “the watch config file endpoint fell over”, which probably would have happened in this outage if 100% of servers went back to watching all of a sudden.

Sure, you could add an error handler for that too, and for prometheus is being slow, and an infinite other things. Or, you could just move process management and reporting out of process.

Re: Cloudflare outage on November 18, 2025 post mortem

#308

Post by mh- »

SilverElfin wrote: post #305
mh- wrote: post #285

Earlier quoted context omitted.

Why give this sort of content more visibility/reach? I'm sure that's not your intent, so I hope my comment gives you an opportunity to reflect on the effects of syndicating such stupidity, no matter what platform it comes from.

Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen a…

Fair points; there's certainly a balance to be struck between raising awareness and amplifying, and I admittedly have no idea where that line is.

Re: Cloudflare outage on November 18, 2025 post mortem

#309

Post by AdieuToLogic »

jcalvinowens wrote: post #275
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

>> This is the multi-million dollar .unwrap() story.

> That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assuming `features.append_with_names` would never fail. Instead, the routine likely should have worked within `Result`.

Re: Cloudflare outage on November 18, 2025 post mortem

#310

Post by spprashant »

A lot of outages off late seem to be related to automated config management.

Companies seem to place a lot of trust is configs being pushed automatically without human review into running systems. Considering how important these configs are, shouldn't they perhaps first be deployed to a staging/isolated network for a monitoring window before pushing to production systems?

Not trying to pontificate here, these systems are more complicated than anything I have maintained. Just trying to think of best practices perhaps everyone can adopt.

Post reply on HN