Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

731–740 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#731

Post by francisofascii »

eastdakota wrote: post #32
lukan wrote: post #20

"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our stat…

We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.

This situation reminds me of risk assessment, where you sometimes assume two rare events are independent, but later learn they are actually highly correlated.

Re: Cloudflare outage on November 18, 2025 post mortem

#732

Post by mmaia »

drc500free wrote: post #666

Makes me wonder which team is responsible for that feature generating query, and if they follow full engineering level QA. It might be deferred to an MLE team that is better than the data scientists but less rigorous than software needs to be.

Exactly. The post screams about all the issues I've seen in multiple companies between DS/MLE and SE/DevOps.

Re: Cloudflare outage on November 18, 2025 post mortem

#733

Post by ljm »

ViewTrick1002 wrote: post #672
tormeh wrote: post #525

Earlier quoted context omitted.

Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code. And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.

What would the difference be if they had enforced no unwraps/expects/slicing and instead logged the error and returned a 500? As the user, I can't tell the difference, but it might have sped up their recovery a bit.

Pretty much - the time spent ruling out the hypothesis that it was a cyberattack would have been time spent investigating the uptick in deliberately written error logs, since you would expect alerts to be triggered if those exceed a threshold.

I imagine it would also require less time debugging a panic. That kind of breadcrumb trail in your logs is a gift to the future engineer and also customers who see a shorter period of downtime.

Re: Cloudflare outage on November 18, 2025 post mortem

#734

Post by echelon »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is the multi-million dollar .unwrap() story.

First multi-million dollar .unwrap() story.

Re: Cloudflare outage on November 18, 2025 post mortem

#735

Post by itzjacki »

huijzer wrote: post #615
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> They failed on so many levels here. That's often the case with human error as especially aviation safety experts know: https://en.wikipedia.org/wiki/Swiss_cheese_model

Exactly. The only way this could happen in the first place was _because_ they failed at so many levels. And as a result, more layers of Swiss cheese will be added, and holes in existing ones will be patched. This process is the reason flying is so safe, and the reason why Cloudflare will be a little bit more resilient tomorrow than it was yesterday.

Re: Cloudflare outage on November 18, 2025 post mortem

#736

Post by echelon »

sznio wrote: post #471
jcalvinowens wrote: post #275

Earlier quoted context omitted.

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

>If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same. If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.

If the `.unwrap()` was replaced with

1:

  ?
2:

  map_err, or, or_else, etc.
3:

  match ... { 
    Ok(..) => {}, 
    Err(..) => {},
  }
4:

  if let ... {
  }
Then it would have been idiomatic Rust code and wouldn't have failed at all.

The function signature returned a `Result`

Seems like it should have returned an Err((ErrorFlags, i32)) here. Case 2 or 3 above would have done nicely.

Removing unwrap() from Rust would have forced the proper handling of the function call and would have prevented this.

Unwrap() is Rust's original sin.

Re: Cloudflare outage on November 18, 2025 post mortem

#737

Post by jandrewrogers »

inferiorhuman wrote: post #521
brabel wrote: post #495

Earlier quoted context omitted.

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

Dunno, I think the alternatives have their own pretty significant downsides. All would require front loading more in-depth understanding of error handling and some would just be quite a bit more verbose. IMO making unwrap a clippy lint (or perhaps a warning) would be a decent start. Or maybe renaming unwrap.

This strikes me as a culture issue more than one of language.

A tenet of systems code is that every possible error must be handled explicitly and exhaustively close to the point of occurrence. It doesn’t matter if it is Rust, C, etc. Knowing how to write systems code is unrelated to knowing a systems language. Rust is a systems language but most people coming into Rust have no systems code experience and are “holding it wrong”. It has been a recurring theme I’ve seen with Rust development in a systems context.

C is pretty broken as a language but one of the things going for it is that it has a strong systems code culture surrounding it that remembers e.g. why we do all of this extra error handling work. Rust really needs systems code practice to be more strongly visible in the culture around the language.

Re: Cloudflare outage on November 18, 2025 post mortem

#738

Post by torginus »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Say what you want exception haters, but at least in exceptions-as-default languages, the decision of a particular issues is fatal to the whole program can be decided centrally at a high level, and not every choice is forced to be up to individual discretion.

Re: Cloudflare outage on November 18, 2025 post mortem

#740

Post by throw0101c »

benjiro wrote: post #723
sofixa wrote: post #595

Earlier quoted context omitted.

> There are many self-hosted alternatives to protect against botnet Whatever you do, unless you have their bandwidth capacity, at some point those "self-hosted" will get flooded with traffic.

As yourself more the question, is your service that important to need 99.999% uptime? Because i get the impression that people are so fixated on this uptime concept, that the idea of being down for a few hours is the most horrible issue in the world. To the point that they rather hand over control of their own system to a 3th party, then accept a downtime. The fact that cloudflare can literally ready every bit of com…

> As yourself more the question, is your service that important to need 99.999% uptime?

What is the cost of many-9s uptime from Cloudflare? For DDoS protection it is $0/month on their free tier:

* https://www.cloudflare.com/en-ca/plans/

Post reply on HN