Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

691–700 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#691

Post by kevincox »

hnarn wrote: post #679
watchful_moose wrote: post #584

Earlier quoted context omitted.

Yep, a decent canary mechanism should have caught this. There's a trade off between canarying and rollout speed, though. If this was a system for fighting bots, I'd expect it to be optimized for the latter.

Presumably optimal rollout speed entails something like or as close to ”push it everywhere all at once and activate immediately” that you can get — that’s fine if you want to risk short downtime rather than delays in rollout, what I don’t understand is why the nodes don’t have any independent verification and rollback mechanism. I might be underestimating the complexity but it really doesn’t sound much more involved…

I think they need to strongly evaluate if they need this level of rollout speed. Even spending a few minutes with an automated canary gives you a ton of safety.

Even if the servers weren't crashing it is possible that a bet set of parameters results in far too many false positives which may as well be complete failure.

Re: Cloudflare outage on November 18, 2025 post mortem

#692

Post by philipwhiuk »

thatoneengineer wrote: post #413

The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen". The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they w…

For unwrap, Cloudflare should consider adding lint tooling that prevents unwrap being added to production code.

Re: Cloudflare outage on November 18, 2025 post mortem

#693

Post by eastdakota »

philipwhiuk wrote: post #296
bayesnet wrote: post #11

Earlier quoted context omitted.

And a well-written one at that. Compared to the AWS port-mortem this could be literature.

Except it fails to document anything about the actions they made to Warp in London during the resolution.

There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.

Re: Cloudflare outage on November 18, 2025 post mortem

#694

Post by antihero »

Ygg2 wrote: post #541
sznio wrote: post #471

Earlier quoted context omitted.

>If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same. If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.

> If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter. It wouldn't, not meaningfully. The outage was caused by change in how they processed the queries. They had no way to observe the changes, nor canaries to see that change is killing them. Plus, they would still need to manually feed and restart services that ingested bad configs. `expect` would…

A billion alerts in DD/Sentry/whatever saying the exact problem that coincide with the exact graph of failures would probably be helpful if someone looked at them.

Re: Cloudflare outage on November 18, 2025 post mortem

#695

Post by vultour »

anal_reactor wrote: post #487

Honestly... everyone shit themselves that internet doesn't work, but next week this outage will be forgotten by 99% of population. I was doing something on my PC when I saw clear information that Cloudflare is down, so I decided to just go take a nap, then read a book, then go for a walk. Once I was done, the internet was working again. Panic was not necessary on my side. What I'm trying to say is that things would b…

It's funny how everyone seems to be having a meltdown over this. I didn't even notice anything was wrong until I read about it on Reddit 5 hours later, even though I was working all day. Sounds to me like people are too reliant on random websites.

Re: Cloudflare outage on November 18, 2025 post mortem

#696

Post by echelon »

wrs wrote: post #416
lenkite wrote: post #408

Earlier quoted context omitted.

What about memory allocation - how will you stop that from panicking ? `Vec::resize` will always panic in Rust. And this is just one example out of thousands in the Rust stdlib. Unless the language addresses no-panic in its governing design or allows try-catch, not sure how you go about this.

That is slowly being addressed, but meanwhile it’s likely you have a reliable upper bound on how much heap your service needs, so it’s a much smaller worry. There are also techniques like up-front or static allocation if you want to make more certain.

I'm far more worried about some dependency calling unwrap() or expect() now.

https://github.com/search?q=unwrap%28%29+language%3ARust&typ...

This is ridiculous. We're probably going to start seeing more of these. This was just the first, big highly visible instance.

We should have a name for this similar to "my code just NPE'd". I suggest "unwrapped", as in, "My Rust app just unwrapped a present."

I think we should start advocating for the deprecation and eventual removal of the unwrap/expect family of methods. There's no reason engineers shouldn't be handling Options and Results gracefully, either passing the state to the caller or turning to a success or fail path. Not doing this is just laziness.

Re: Cloudflare outage on November 18, 2025 post mortem

#697

Post by ozgrakkurt »

VMG wrote: post #654
csomar wrote: post #501

Earlier quoted context omitted.

It doesn’t disappear, it forces you to handle it.

Propagating upwards a valid way of handling it and often the correct answer. There needs to be something at the top level that can handle a crashing process.

You mean like kubernetes that restarts your program when it crashes?

Re: Cloudflare outage on November 18, 2025 post mortem

#698

Post by an_ko »

echelon wrote: post #655
milliams wrote: post #600

Earlier quoted context omitted.

[lints.clippy] dbg_macro = "deny" unwrap_used = "deny" expect_used = "deny"

https://github.com/search?q=unwrap%28%29+language%3ARust&typ... This is sobering. My new fear is some dependency unwrap()ing or expect()ing something where they didn't prove the correctness. Unwrap() and expect() are an anti-pattern and have no place in idiomatic Rust code. The language should move to deprecate them.

I use unwrap a lot, and my most frequent target is unwrapping the result of Mutex::lock. Most applications have no reasonable way to recover from lock poisoning, so if I were forced to write a match for each such use site to handle the error case, the handler would have no choice but to just call panic anyway. Which is equivalent to unwrap, but much more verbose.

Perhaps it needs a scarier name, like "assume_ok".

Re: Cloudflare outage on November 18, 2025 post mortem

#699

Post by pornel »

abigailphoebe wrote: post #289

kudos to getting this blog post out so fast, it’s well written and is appreciated. i’m a little confused on how this was initially confused for an attack though? is there no internal visibility into where 5xx’s are being thrown? i’m surprised there isn’t some kind of "this request terminated at the " error mapping that could have initially pointed you guys towards that over an attack. also a bit taken aback that .unw…

1. Cloudflare is in the business of being a lightning rod for large and targeted DoS attacks. A lot of cases are attacks.

2. Attacks that make it through the usual defences make servers run at rates beyond their breaking point, causing all kinds of novel and unexpected errors.

Additionally, attackers try to hit endpoints/features that amplify severity of their attack by being computationally expensive, holding a lock, or trigger an error path that restarts a service — like this one.

Re: Cloudflare outage on November 18, 2025 post mortem

#700

Post by quotemstr »

wrs wrote: post #86
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

> people have a blind spot for unwrap

It's not about whether you should ban unwrap() in production. You shouldn't. Some errors are logic bugs beyond which a program can't reasonably continue. The problem is that the language makes it too easy for junior developers (and AI!) to ignore non-logic-bug problems with unwrap().

Programmers early in their careers will do practically anything to avoid having to think about errors and they get angry when you tell them about it.

Post reply on HN