Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

801–810 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#801

Post by AniseAbyss »

jve wrote: post #582
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> I'm migrating my customers off Cloudflare. Is that an overreaction? Name me global, redundant systems that have not (yet) failed. And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic. I mean no service have 100% uptime - just that some have more nines than others.

[dead]

Re: Cloudflare outage on November 18, 2025 post mortem

#802

Post by underdeserver »

burntsushi wrote: post #784
underdeserver wrote: post #751

Earlier quoted context omitted.

Eh, "mistaken" might be a bit harsh. He's stating an opinion, which you and I disagree with. > The language should grow the ability to mark this code as dangerous, and we should have static tools to exclude this code from our dependency tree. Might be useful to point out that this static tool exists (clippy::unwrap_used).

They said: > unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features. That's factually incorrect. (And insulting.)

I disagree with that characterization. Using unwrap() like you suggest in your blog post is an intentional, well-thought-out choice. Using unwrap() the way Cloudflare did it is, with hindsight, a bad choice, that doesn't utilize the language's design features.

Note that they're not criticizing the language. I read "Rust developers" in this context as developers using Rust, not those who develop the language and ecosystem. (In particular they were not criticizing you.)

I think it's reasonable to question the use of unwrap() in this context. Taking a cue from your blog post^ under runtime invariant violations, I don't think this use matches any of your cases. They assumed the size of a config file is small, it wasn't, so the internet crashed.

^ https://burntsushi.net/unwrap/#what-is-my-position

Re: Cloudflare outage on November 18, 2025 post mortem

#803

Post by cocoa19 »

gucci-on-fleek wrote: post #70
eastdakota wrote: post #35

Earlier quoted context omitted.

Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.

Thanks for the explanation! This definitely reminds me of CrowdStrike outages last year: - A product depends on frequent configuration updates to defend against attackers. - A bad data file is pushed into production. - The system is unable to easily/automatically recover from bad data files. (The CrowdStrike outages were quite a bit worse though, since it took down the entire computer and remediation required manual…

It might remind you of Crowdstrike because of the scale.

Outages are in a large majority of cases caused by change, either deployments of new versions or configuration changes.

Re: Cloudflare outage on November 18, 2025 post mortem

#804

Post by burntsushi »

underdeserver wrote: post #802
burntsushi wrote: post #784

Earlier quoted context omitted.

They said: > unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features. That's factually incorrect. (And insulting.)

I disagree with that characterization. Using unwrap() like you suggest in your blog post is an intentional, well-thought-out choice. Using unwrap() the way Cloudflare did it is, with hindsight, a bad choice, that doesn't utilize the language's design features. Note that they're not criticizing the language. I read "Rust developers" in this context as developers using Rust, not those who develop the language and ecosy…

The commenter also said:

> We shouldn't be using unwrap() or expect() at all.

So the context of their comment is not some specific nuanced example. They made a blanket statement.

> Note that they're not criticizing the language. I read "Rust developers" in this context as developers using Rust, not those who develop the language and ecosystem.

I have the same interpretation.

> I think it's reasonable to question the use of unwrap() in this context. Taking a cue from your blog post^ under runtime invariant violations, I don't think this use matches any of your cases. They assumed the size of a config file is small, it wasn't, so the internet crashed.

Yes? I didn't say it wasn't reasonable to question the use of unwrap() here. I don't think we really have enough information to know whether it was inappropriate or not.

unwrap() is all about nuance. I hope my blog post conveyed that. Because unwrap() is a manifestation of an assertion on a runtime invariant. A runtime invariant can be arbitrarily complicated. So saying things like, "we shouldn't be using unwrap() or expect() at all" is an extreme position to carve out that is also way too generalized.

I stand by what I said. They are factually mistaken in their characterization of the use of unwrap()/expect() in general.

Re: Cloudflare outage on November 18, 2025 post mortem

#805

Post by groundzeros2015 »

frumplestlatz wrote: post #792
groundzeros2015 wrote: post #790

Earlier quoted context omitted.

Most properties of programs cannot be validated at compile time and must be checked at runtime. But you’re still missing it. Crashing is not bad. It’s good. It’s how you leverage OS level security and reliability.

This wasn't a runtime property that could not be validated at compile time. And you don't need to fall back on "OS level security and reliability" when your type system is enforcing an application-level invariants. In fact I'd argue that crashing is bad. It means you failed to properly enumerate and express your invariants, hit an unanticipated state, and thus had to fail in a way that requires you to give up and fal…

> The end result was a massive world-wide outage.

The world wide outage was actually caused by deploying several incorrect programs in an incorrect system.

The root one was actually a bad query as outlined in the article.

Let’s get philosophical for a second. Programs WILL be written incorrectly - you will deploy to production something that can’t possibly work. What should you do with a program that can’t work? Pretend this can’t happen? Or let you know so you can fix it?

Re: Cloudflare outage on November 18, 2025 post mortem

#806

Post by underdeserver »

burntsushi wrote: post #804
underdeserver wrote: post #802

Earlier quoted context omitted.

I disagree with that characterization. Using unwrap() like you suggest in your blog post is an intentional, well-thought-out choice. Using unwrap() the way Cloudflare did it is, with hindsight, a bad choice, that doesn't utilize the language's design features. Note that they're not criticizing the language. I read "Rust developers" in this context as developers using Rust, not those who develop the language and ecosy…

The commenter also said: > We shouldn't be using unwrap() or expect() at all. So the context of their comment is not some specific nuanced example. They made a blanket statement. > Note that they're not criticizing the language. I read "Rust developers" in this context as developers using Rust, not those who develop the language and ecosystem. I have the same interpretation. > I think it's reasonable to question the…

> So the context of their comment is not some specific nuanced example. They made a blanket statement.

That is their opinion, I disagree with it, but I don't think it's an insulting or invalid opinion to have. There are codebases that ban nulls in other languages too.

> They are factually mistaken in their characterization of the use of unwrap()/expect() in general.

It's an opinion about a stylistic choice. I don't see what fact there is here that could be mistaken.

Re: Cloudflare outage on November 18, 2025 post mortem

#807

Post by dylan604 »

globalise83 wrote: post #758
tetec1 wrote: post #59

Earlier quoted context omitted.

Yeah, I can imagine that this insertion was some high-pressure job.

The computer science equivalent of choosing between the red, green and blue wires when disarming a nuke with 15 seconds left on the clock

Is it though? Or is it, oh, this is such a simple change that we really don't need to test it attitude? I'm not saying this applies to TFA, but some people are so confident that no pressure is felt.

However, you forgot that the lighting conditions are where only red lights from the klaxons are showing so you really can't differentiate the colors of the wires

Re: Cloudflare outage on November 18, 2025 post mortem

#808

Post by nmadden »

groundzeros2015 wrote: post #785
philipwhiuk wrote: post #692

Earlier quoted context omitted.

For unwrap, Cloudflare should consider adding lint tooling that prevents unwrap being added to production code.

It’s a feature, not a bug. Assert assumptions and crash on bad one. Crashing is not an outage. It’s a restart and a stack trace for you to fix.

> Crashing is not an outage.

Are you in the right thread?

Re: Cloudflare outage on November 18, 2025 post mortem

#809

Post by empath75 »

groundzeros2015 wrote: post #785
philipwhiuk wrote: post #692

Earlier quoted context omitted.

For unwrap, Cloudflare should consider adding lint tooling that prevents unwrap being added to production code.

It’s a feature, not a bug. Assert assumptions and crash on bad one. Crashing is not an outage. It’s a restart and a stack trace for you to fix.

Even if you want it to crash, you almost never unwrap. At the very least you would use .expect() so you get a reasonable error message -- or even better you handle the potential error.

Re: Cloudflare outage on November 18, 2025 post mortem

#810

Post by wongarsu »

underdeserver wrote: post #802
burntsushi wrote: post #784

Earlier quoted context omitted.

They said: > unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features. That's factually incorrect. (And insulting.)

I disagree with that characterization. Using unwrap() like you suggest in your blog post is an intentional, well-thought-out choice. Using unwrap() the way Cloudflare did it is, with hindsight, a bad choice, that doesn't utilize the language's design features. Note that they're not criticizing the language. I read "Rust developers" in this context as developers using Rust, not those who develop the language and ecosy…

Echelon's comment was "We shouldn't be using unwrap() or expect() at all. [...] unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers". Even in my most generous interpretation I can't see how that is anything except a rejection of all unwraps (and equivalent constructs like expect()).

I fully agree with burntsushi that echelon is taking an extreme and arguably wrong stance. His sentiment becomes more and more correct as Rust continues to evolve ways to avoid unwrap as an ergonomic shortcut, but I don't think we are quite there yet for general use. There absolutely is code that should never panic, but that involves tradeoffs and design choices that aren't true for every project (or even the majority of them)

Post reply on HN