Cloudflare outage on November 18, 2025 post mortem • Hacker News Board

Re: Cloudflare outage on November 18, 2025 post mortem

#811

Post by burntsushi » Wed, Nov 19, 2025, 3:57 PM UTC

underdeserver wrote: post #806

burntsushi wrote: post #804

Earlier quoted context omitted.

The commenter also said: > We shouldn't be using unwrap() or expect() at all. So the context of their comment is not some specific nuanced example. They made a blanket statement. > Note that they're not criticizing the language. I read "Rust developers" in this context as developers using Rust, not those who develop the language and ecosystem. I have the same interpretation. > I think it's reasonable to question the…

> So the context of their comment is not some specific nuanced example. They made a blanket statement. That is their opinion, I disagree with it, but I don't think it's an insulting or invalid opinion to have. There are codebases that ban nulls in other languages too. > They are factually mistaken in their characterization of the use of unwrap()/expect() in general. It's an opinion about a stylistic choice. I don't s…

I'm finding this exchange frustrating, and now we're going in circles. I'll say this one last time in as clear language as I can. They said this:

> unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features.

The factually incorrect part of this is the statement that use of `unwrap()`, `expect()` and so on is caused by X or Y, where X is "lazy Rust developers" and Y is "Rust developers not utilizing the language's design features." But there are, factually, other causes than X or Y for use of `unwrap()`, `expect()` and so on. So stating that it is all caused by X or Y is factually incorrect. Moreover, X is 100% insulting when applied to any one specific individual. Y can be insulting when applied to any one specific individual.

Now this:

> We shouldn't be using unwrap() or expect() at all.

That's an opinion. It isn't factually incorrect. And it isn't insulting.

Re: Cloudflare outage on November 18, 2025 post mortem

#812

Post by QREguy » Wed, Nov 19, 2025, 4:02 PM UTC

eastdakota wrote: post #722

keypusher wrote: post #326

The most surprising thing to me here is that it took 3 hours to root cause, and points to a glaring hole in the platform observability. Even taking into account the fact that the service was failing intermittently at first, it still took 1.5 hours after it started failing consistently to root cause. But the service was crashing on startup. If a core service is throwing a panic at startup like that, it should be raisi…

That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing wi…

Any "limits" on system should be alerted... like at 70% or 80% threshold.. it might be worth it for a SRE to revisit the system limits and ensuring threshold based alerting around it..

Re: Cloudflare outage on November 18, 2025 post mortem

#813

Post by asa400 » Wed, Nov 19, 2025, 4:03 PM UTC

tuetuopay wrote: post #619

andy_ppp wrote: post #603

Earlier quoted context omitted.

This is why the Erlang/Elixir methodology of having supervision and letting things crash gracefully is so useful. You can either handle every single error gracefully or handle crashing gracefully - it's much easier and more realistic in large codebases to do the later.

This would not have helped: the code would crash before doing anything useful at all. If anything, the "crash early" mentality may even be nefarious: instead of handling the error and keeping the old config, you would spin on trying to load a broken config on startup.

Continuing only makes sense for cases you know you can handle.

_In theory_ they could have used the old config, but maybe there are reasons that’s not possible in Cloudflare’s setup. Whether or not that’s an invariant violation or just an error that can be handled and recovered from is a matter of opinion in system design.

And crashing on an invariant violation is exactly the right thing to do rather than proceed in an undefined state.

Re: Cloudflare outage on November 18, 2025 post mortem

#814

Post by ChrisMarshallNY » Wed, Nov 19, 2025, 4:04 PM UTC

groundzeros2015 wrote: post #786

ChrisMarshallNY wrote: post #190

Earlier quoted context omitted.

Swift has implicit unwrap (!), and explicit unwrap (?). I don't like to use implicit unwrap. Even things that are guaranteed to be there, I treat as explicit (For example, (self.view?.isEnabled ?? false) , in a view controller, instead of self.view.isEnabled ). I always redefine @IBOutlets from: @IBOutlet weak var someView! to: @IBOutlet weak var someView? I'm kind of a "belt & suspenders" type of guy.

This is terrible. The whole reason they introduced this is because IBOutlets would get silently disconnected and then in the field a user would complain that a feature stopped working. Crash early, crash often. Find the bugs and bad assumptions.

> This is terrible.

No it's not. Read my other comments.

ToMAYto, ToMAHto.

I have learned that it's a bad idea to trash other folks' methodologies without taking the time to understand why they do things, the way they do.

I have found dogma to be an impediment, in my own work. As I've gotten older, the sharp edges have been sanded off.

Have a great day!

Re: Cloudflare outage on November 18, 2025 post mortem

#815

Post by dist1ll » Wed, Nov 19, 2025, 4:09 PM UTC

hansvm wrote: post #793

dist1ll wrote: post #287

Earlier quoted context omitted.

Sure, these days I'm mostly working on a few compilers. Let's say I want to make a fixed-size SSA IR. Each instruction has an opcode and two operands (which are essentially pointers to other instructions). The IR is populated in one phase, and then lowered in the next. During lowering I run a few peephole and code motion optimizations on the IR, and then do regalloc + asm codegen. During that pass the IR is mutated a…

One normal "trick" is phantom typing. You create a type representing indices and have a small, well-audited portion of unsafe code handling creation/unpacking, where the rest of the code is completely safe. The details depend a lot on what you're doing and how you're doing it. Does the graph grow? Shrink? Do you have more than one? Do you care about programmer error types other than panic/UB? Suppose, e.g., that your…

I do use a combination of newtyped indices + singleton arenas for data structures that only grow (like the AST). But for the IR, being able to remove nodes from the graph is very important. So phantom typing wouldn't work in that case.

Re: Cloudflare outage on November 18, 2025 post mortem

#816

Post by jjice » Wed, Nov 19, 2025, 4:09 PM UTC

There's (obviously) a lot of discussion around the use of `unwrap` in production code. I feel like I'm watching comments speak past each other right now.

I'd agree that the use of `unwrap` could possibly make sense in a place where you do want the system to fail hard. There's lot of good reasons to make the system fail hard. I'd lean towards an `expect` here, but whatever.

That said, the function already returns a `Result` and we don't know what the calling code looks like. Maybe it does do an `unwrap` there too, or maybe there is a save way for this to log and continue that we're not aware of because we don't have enough info.

Should a system as critical as the CF proxy fail hard? I don't know. I'd say yes if it was the kind of situation that could revert itself (like an incremental rollout), but this is such an interesting situation since it's a config being rolled out. Hindsight is 20:20 obviously, but it feels like there should've been better logging, deployment, rollback, and parsing/validation capabilities, no matter what the `unwrap`/`Result` option is.

Also, it seems like the initial Clickhouse changes could've been testing much better, but I'm sure the CF team realizes that.

On the bright side, this is a very solid write up so quickly after the outage. Much better than those times we get it two weeks later.

Re: Cloudflare outage on November 18, 2025 post mortem

#817

Post by groundzeros2015 » Wed, Nov 19, 2025, 4:11 PM UTC

nmadden wrote: post #808

groundzeros2015 wrote: post #785

Earlier quoted context omitted.

It’s a feature, not a bug. Assert assumptions and crash on bad one. Crashing is not an outage. It’s a restart and a stack trace for you to fix.

> Crashing is not an outage. Are you in the right thread?

Did you skip all the other context about the other systems that failed?

The problem was a query producing incorrect data. The crash helped them find it.

What do you think happens when a program crashes?

Re: Cloudflare outage on November 18, 2025 post mortem

#818

Post by groundzeros2015 » Wed, Nov 19, 2025, 4:14 PM UTC

ChrisMarshallNY wrote: post #814

groundzeros2015 wrote: post #786

Earlier quoted context omitted.

This is terrible. The whole reason they introduced this is because IBOutlets would get silently disconnected and then in the field a user would complain that a feature stopped working. Crash early, crash often. Find the bugs and bad assumptions.

> This is terrible. No it's not. Read my other comments. ToMAYto, ToMAHto. I have learned that it's a bad idea to trash other folks' methodologies without taking the time to understand why they do things, the way they do. I have found dogma to be an impediment, in my own work. As I've gotten older, the sharp edges have been sanded off. Have a great day!

> without taking the time to understand why they do things, the way they do.

Oh I am aware. They do it because

A) they don’t have a mental model of correct execution. Events just happen to them with a feeling of powerlessness. So rather than trying to form one they just litter the code with cases things that might happen

> As I've gotten older, the sharp edges have been sanded off.

B) they have grown in bad organizations with bad incentives that penalize the appearance of making mistakes. So they learn to hide them.

For example there might be an initiative that rewards removing crashes in favor of silent error.

Re: Cloudflare outage on November 18, 2025 post mortem

#819

Post by underdeserver » Wed, Nov 19, 2025, 4:15 PM UTC

burntsushi wrote: post #811

underdeserver wrote: post #806

Earlier quoted context omitted.

> So the context of their comment is not some specific nuanced example. They made a blanket statement. That is their opinion, I disagree with it, but I don't think it's an insulting or invalid opinion to have. There are codebases that ban nulls in other languages too. > They are factually mistaken in their characterization of the use of unwrap()/expect() in general. It's an opinion about a stylistic choice. I don't s…

I'm finding this exchange frustrating, and now we're going in circles. I'll say this one last time in as clear language as I can. They said this: > unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features. The factually incorrect part of this is the statement that use of `unwrap()`, `expect()` and so on is caused by X or Y, where X…

I'm sorry I'm frustrating you. It was not my intention. For what it's worth, I use ripgrep every day, and it's made my life appreciably better. (Same goes for Astral products.) Thank you for that, and I wish your day improves.

> unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features

I just read that line as shorthand for large outages caused by misuse of unwrap(), expect(), bad math etc. - all caused by...

That's also an opinion, by my reading.

I assumed we were talking specifically about misuses, not all uses of unwrap(), or all bad bugs. Anyway, I think we're ultimately saying the same thing. It's ironic in its own way.

Re: Cloudflare outage on November 18, 2025 post mortem

#820

Post by tuetuopay » Wed, Nov 19, 2025, 4:25 PM UTC

asa400 wrote: post #813

tuetuopay wrote: post #619

Earlier quoted context omitted.

This would not have helped: the code would crash before doing anything useful at all. If anything, the "crash early" mentality may even be nefarious: instead of handling the error and keeping the old config, you would spin on trying to load a broken config on startup.

Continuing only makes sense for cases you know you can handle. _In theory_ they could have used the old config, but maybe there are reasons that’s not possible in Cloudflare’s setup. Whether or not that’s an invariant violation or just an error that can be handled and recovered from is a matter of opinion in system design. And crashing on an invariant violation is exactly the right thing to do rather than proceed in…

Given the context and what the configuration file contains, I'd argue it's mission-critical for the software to keep running with the previous configuration. Otherwise you might shutdown the internet. Honestly, I'm pretty sure their pre-rewrite version had such logic, and it was forgotten or still on the TODO pile for the rewrite version.

At a previous job (cloud provider), we've had exactly this kind of issue, with exactly the same root cause. The entrypoint for the whole network had a set of rules (think a NAT gateway) that were reloaded periodically from the database. Someone rewrote that bit of plumbing from Python to Go. Someone else performed a database migration. Suddenly, the plumbing could not find the data, and pushed an empty file to prod. The rewrite lacked "if empty, do nothing and raise an alert", that the previous one had. I'll let you imagine what happened next :)