Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

781–790 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#781

Post by mike_hearn »

bombela wrote: post #770
mike_hearn wrote: post #760

Earlier quoted context omitted.

Really not! This is a huge faceplant for writing things in Rust. If they had been writing their code in Java/Kotlin instead of Rust, this outage either wouldn't have happened at all (a failure to load a new config would have been caught by a defensive exception handler), or would have been resolved in minutes instead of hours. The most useful thing exceptions give you is not static compile time checking, it's the sta…

https://doc.rust-lang.org/std/backtrace/index.html#environme... tldr: Capturing a backtrace can be a quite expensive runtime operation, so the environment variables allow either forcibly disabling this runtime performance hit or allow selectively enabling it in some programs. By default it is disabled in release mode.

It's one of the problems with using result types. You don't distinguish between genuinely exceptional events and things that are expected to happen often on hot paths, so the runtime doesn't know how much data to collect.

Re: Cloudflare outage on November 18, 2025 post mortem

#782

Post by the8472 »

echelon wrote: post #651
bigstrat2003 wrote: post #347

Earlier quoted context omitted.

unwrap is explicit.

What happens if a dependency of mine unwrap()s or expect()s something that is an error or isn't there? What should I do? How was I informed as a user? It's not in the type signature. Sounds like I get to indeterminately crash at runtime and have a fun time debugging.

That would require an effects system[0] like Koka's[1]. Then one could not only express the absence of panics but also allocations, infinite loops and various other undesirable effects within some call-trees. This is a desirable feature, but an enormous undertaking.

Absent that there are hacks like no_panic[2]

[0] https://blog.yoshuawuyts.com/extending-rusts-effect-system/ [1] https://koka-lang.github.io/koka/doc/book.html#why-effects [2] https://crates.io/crates/no-panic

Re: Cloudflare outage on November 18, 2025 post mortem

#783

Post by keypusher »

eastdakota wrote: post #722
keypusher wrote: post #326

The most surprising thing to me here is that it took 3 hours to root cause, and points to a glaring hole in the platform observability. Even taking into account the fact that the service was failing intermittently at first, it still took 1.5 hours after it started failing consistently to root cause. But the service was crashing on startup. If a core service is throwing a panic at startup like that, it should be raisi…

That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing wi…

Thank you for the clarification and insight, with that context it does make more sense to me. Is there anything you think can be done to improve the ability to identify issues like this more quickly in the future?

Re: Cloudflare outage on November 18, 2025 post mortem

#784

Post by burntsushi »

underdeserver wrote: post #751
burntsushi wrote: post #727

Earlier quoted context omitted.

I'm on the Rust libs-api team and you're mistaken. I use `unwrap()` all the time. My blog on this topic was linked above, you should read it: https://burntsushi.net/unwrap/

Eh, "mistaken" might be a bit harsh. He's stating an opinion, which you and I disagree with. > The language should grow the ability to mark this code as dangerous, and we should have static tools to exclude this code from our dependency tree. Might be useful to point out that this static tool exists (clippy::unwrap_used).

They said:

> unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features.

That's factually incorrect. (And insulting.)

Re: Cloudflare outage on November 18, 2025 post mortem

#785

Post by groundzeros2015 »

philipwhiuk wrote: post #692
thatoneengineer wrote: post #413

The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen". The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they w…

For unwrap, Cloudflare should consider adding lint tooling that prevents unwrap being added to production code.

It’s a feature, not a bug. Assert assumptions and crash on bad one.

Crashing is not an outage. It’s a restart and a stack trace for you to fix.

Re: Cloudflare outage on November 18, 2025 post mortem

#786

Post by groundzeros2015 »

ChrisMarshallNY wrote: post #190
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Swift has implicit unwrap (!), and explicit unwrap (?). I don't like to use implicit unwrap. Even things that are guaranteed to be there, I treat as explicit (For example, (self.view?.isEnabled ?? false) , in a view controller, instead of self.view.isEnabled ). I always redefine @IBOutlets from: @IBOutlet weak var someView! to: @IBOutlet weak var someView? I'm kind of a "belt & suspenders" type of guy.

This is terrible. The whole reason they introduced this is because IBOutlets would get silently disconnected and then in the field a user would complain that a feature stopped working.

Crash early, crash often. Find the bugs and bad assumptions.

Re: Cloudflare outage on November 18, 2025 post mortem

#787

Post by frumplestlatz »

branko_d wrote: post #780
frumplestlatz wrote: post #556

Earlier quoted context omitted.

> Throwing an exception does not necessarily mean that your program is suddenly in an unsupported state When everyone uses runtime exceptions and doesn’t count for exception handling in every possible code path, that’s exactly what it means.

Sure, but the same is true of any error handling strategy. When you work with exceptions, the key is to assume that every line can throw unless proven otherwise, which in practice means almost all lines of code can throw. Once you adopt that mental model, things get easier.

Explicit error handling strategies allow you to not worry about all the code paths that explicitly cannot throw -- which is a lot of them. It makes life a lot easier in the non-throwing case, and doesn't complicate life any more in the throwing case as compared to exception-based error handling.

It also makes errors part of the API contract, which is where they belong, because they are.

Re: Cloudflare outage on November 18, 2025 post mortem

#788

Post by burntsushi »

frumplestlatz wrote: post #765
burntsushi wrote: post #729

Earlier quoted context omitted.

The blog linked in the GP anticipates this rebuttal and already addresses it. Your argument also implies that things like `slice[i]` are never okay.

`slice[i]` is also a hole in the type system, but at least it’s generally relying on a local invariant, immediate to the surrounding context, that does not require lying about invariants across your API surface. The blog post doesn’t address the issue, it simply pretends it’s not a real problem. Also from the post: “If we were to steelman advocates in favor of this style of coding, then I think the argument is probab…

`slice[i]` is just sugar for `slice.get(i).unwrap()`. And whether it's a "local" invariant or not is orthogonal. And `unwrap()` does not "require lying about invariants across your API surface."

> The blog post doesn’t address the issue, it simply pretends it’s not a real problem.

It very explicitly addresses it! It even gives real examples.

> Also from the post: “If we were to steelman advocates in favor of this style of coding, then I think the argument is probably best limited to certain high reliability domains. I personally don’t have a ton of experience in said domains …” > > Enough said.

Ad hominem... I don't have experience working on, e.g., medical devices upon which someone's life depends. So the point of that sentence is to say, "yes, I acknowledge this advice may not apply there." You also cherry picked that quote and left off the context, which is relevant here.

And note that you said:

> I have to disagree that unwrap is ever OK.

That's an extreme position. It isn't caveated to only apply to certain contexts.

Re: Cloudflare outage on November 18, 2025 post mortem

#789

Post by frumplestlatz »

groundzeros2015 wrote: post #785
philipwhiuk wrote: post #692

Earlier quoted context omitted.

For unwrap, Cloudflare should consider adding lint tooling that prevents unwrap being added to production code.

It’s a feature, not a bug. Assert assumptions and crash on bad one. Crashing is not an outage. It’s a restart and a stack trace for you to fix.

The type system is for asserting assumptions like "this cannot fail". You don't crash at all.

Re: Cloudflare outage on November 18, 2025 post mortem

#790

Post by groundzeros2015 »

frumplestlatz wrote: post #789
groundzeros2015 wrote: post #785

Earlier quoted context omitted.

It’s a feature, not a bug. Assert assumptions and crash on bad one. Crashing is not an outage. It’s a restart and a stack trace for you to fix.

The type system is for asserting assumptions like "this cannot fail". You don't crash at all.

Most properties of programs cannot be validated at compile time and must be checked at runtime.

But you’re still missing it. Crashing is not bad. It’s good. It’s how you leverage OS level security and reliability.

Post reply on HN