Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

351–360 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#351

Post by tempay »

WD-42 wrote: post #267
AgentME wrote: post #214

Earlier quoted context omitted.

This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a n…

Yea, Rust is safe but it’s not magic. However Nginx doesn’t panic on malformed config. It exits with hopefully a helpful error code and message. The question is then could the cloudflare code have exited cleanly in a way that made recovery easier instead of just straight panicking.

Would expect with a message meet that criteria of exiting with a more helpful error message? From the postmortem it seems to me like they just didn’t know it even was panicing

Re: Cloudflare outage on November 18, 2025 post mortem

#352

Post by cvhc »

kibwen wrote: post #273
cvhc wrote: post #122

Earlier quoted context omitted.

Some languages and style guides simply forbid throwing exceptions without catching / proper recovery. Google C++ bans exceptions and the main mechanism for propogating errors is `absl::Status` which the caller has to check. Not familiar with Rust but it seems unwrap is such a thing that would be banned.

> Not familiar with Rust but it seems unwrap is such a thing that would be banned. Panics aren't exceptions, any "panic" in Rust can be thought of as an abort of the process (Rust binaries have the explicit option to implement panics as aborts). Companies like Dropbox do exactly this in their similar Rust-based systems, so it wouldn't surprise me if Cloudflare does the same. "Banning exceptions" wouldn't have done an…

Yeah I know but isn't unwrap basically a trivial way to: (1) give up catching the exception/error (the E part in `Result`) that the callee throws; and also (2) escalate it to the point that nothing can catch it. It has such a benign name.

Re: Cloudflare outage on November 18, 2025 post mortem

#353

Post by bri3d »

spprashant wrote: post #298
bri3d wrote: post #260

Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random…

I have nowhere near the experience managing such complex systems, but I can empathize with this. In a high-pressure situations the most obvious things get missed. If someone is convinced System X is at fault, your mind can make leaps to justify every other degraded system is a downstream effect of that. Cause and effect can get switched. Sometimes you have smart people in the room who dig deeper and fish it out, but…

I have plenty of empathy, having been in plenty of similar situations. It's not a matter of "I can't BELIEVE it took that long" (although it is a bit surprising) so much as that I disagree with the key takeaways here in the HN comments section and in the blog itself, which focus strongly on fixing rare edge case issues (the bad ClickHouse query and a bad config file causing a panic via unwrap), rather than reducing MTTR for all issues by improving the debug and monitoring experience.

I'm also suspicious that

> Eliminating the ability for core dumps or other error reports to overwhelm system resources

from the blog had a lot more to do with the issue than perhaps the narrative is letting on.

Re: Cloudflare outage on November 18, 2025 post mortem

#354

Post by bri3d »

reassess_blind wrote: post #312
bri3d wrote: post #260

Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random…

Once they figured it out they didn't have a way to load in a new feature file, had to figure that out, and then restart every machine.

This took one of the three hours; it seems to have taken from 11:28 to 13:37 to recognize that the configuration file panic was the cause of the issue.

Re: Cloudflare outage on November 18, 2025 post mortem

#355

Post by 10000truths »

dist1ll wrote: post #225
10000truths wrote: post #189

Earlier quoted context omitted.

Yes? Funnily enough, I don't often use indexed access in Rust. Either I'm looping over elements of a data structure (in which case I use iterators), or I'm using an untrusted index value (in which case I explicitly handle the error case). In the rare case where I'm using an index value that I can guarantee is never invalid (e.g. graph traversal where the indices are never exposed outside the scope of the traversal),…

If that's the case then hats off. What you're describing is definitely not what I've seen in practice. In fact, I don't think I've ever seen a crate or production codebase that documents infallibility of every single slice access. Even security-critical cryptography crates that passed audits don't do that. Personally, I found it quite hard to avoid indexing for graph-heavy code, so I'm always on the lookout for inter…

My rule of thumb is that unchecked access is okay in scenarios where both the array/map and the indices/keys are private implementation details of a function or struct, since an invariant is easy to manually verify when it is tightly scoped as such. I've seen it used it in:

* Graph/tree traversal functions that take a visitor function as a parameter

* Binary search on sorted arrays

* Binary heap operations

* Probing buckets in open-addressed hash tables

Re: Cloudflare outage on November 18, 2025 post mortem

#356

Post by wrs »

echelon wrote: post #327
kibwen wrote: post #323

Earlier quoted context omitted.

The `unsafe` keyword means something specific in Rust, and panicking isn't unsafe by Rust's definition. Sometimes avoiding partial functions just isn't feasible, and an unwrap (or whatever you want to call the method) is a way of providing a (runtime-checked) proof to the compiler that the function is actually total.

Panics should be explicit, not implicit. unwrap() should effectively work as a Result where the user must manually invoke a panic in the failure branch. Make special syntax if a match and panic is too much boilerplate. This is like an implicit null pointer exception that cannot be statically guarded against. I want a way to statically block any crates doing this from my dependency chain.

Not sure what you're saying with the "work as a Result" part...unwrap is a method on Result. I think you're just saying the unwrap/expect methods should be eliminated?

Re: Cloudflare outage on November 18, 2025 post mortem

#357

Post by wrs »

speed_spread wrote: post #264
wrs wrote: post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

Pet peeve: unwrap() should be deprecated and renamed or_panic(). More consistent with the rest of stdlib methods and appropriately scarier.

That's kind of what I'm saying with the blind spot comment. The words "unwrap" and "expect" should be just as much a scary red flag as the word "panic", but for some reason it seems a lot of people don't see them that way.

Re: Cloudflare outage on November 18, 2025 post mortem

#358

Post by monocularvision »

ChrisMarshallNY wrote: post #190
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Swift has implicit unwrap (!), and explicit unwrap (?). I don't like to use implicit unwrap. Even things that are guaranteed to be there, I treat as explicit (For example, (self.view?.isEnabled ?? false) , in a view controller, instead of self.view.isEnabled ). I always redefine @IBOutlets from: @IBOutlet weak var someView! to: @IBOutlet weak var someView? I'm kind of a "belt & suspenders" type of guy.

So what happens if it ends up being nil? How does your app react?

In this particular case, I would rather crash. It’s easier to spot in a crash report and you get a nice stack trace.

Silent failure is ultimately terrible for users.

Note: for the things I control I try to very explicitly model state in such a way as I never need to force unwrap at all. But for things beyond my control like this situation, I would rather end the program than continue with a state of the world I don’t understand.

Re: Cloudflare outage on November 18, 2025 post mortem

#360

Post by nurettin »

I can never get used to the error happening at call site rather than within the function where the early return of Err happened. It is not "much cleaner", you have no idea which line and file caused it at call site. By default Returning should have a way of setting a marker which can then be used to map back to the line() and file(). 10+ years and still no ergonomics.
Post reply on HN