Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

541–550 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#541

Post by Ygg2 »

sznio wrote: post #471
jcalvinowens wrote: post #275

Earlier quoted context omitted.

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

>If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same. If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.

> If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.

It wouldn't, not meaningfully. The outage was caused by change in how they processed the queries. They had no way to observe the changes, nor canaries to see that change is killing them. Plus, they would still need to manually feed and restart services that ingested bad configs.

`expect` would shave a few minutes; you would still spend hours figuring out and fixing it.

Granted, using expect is better, but it's not a silver bullet.

Re: Cloudflare outage on November 18, 2025 post mortem

#543

Post by xnotcursed »

HL33tibCe7 wrote: post #494

An unwrap like that in production code on the critical path is very surprising to me. I haven’t worked in Rust codebases, but I have never worked in a Go codebase where a `panic` in such a location would make it through code review. Is this normal in Rust?

absolutely not normal, this is why in my opinion it took them so long to understand the core issue. instead of a nice error message and a backtrace saying something like "failed to parse config feature names" they thought they were under attack because the service was just crashing instead.

Re: Cloudflare outage on November 18, 2025 post mortem

#544

Post by chrismorgan »

gucci-on-fleek wrote: post #13

> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was tri…

> much better than their completely false “checking the security of your connection” message

The exact wording (which I can easily find, because a good chunk of the internet gives it to me, because I’m on Indian broadband):

> example.com needs to review the security of your connection before proceeding.

It bothers me how this bald-faced lie of a wording has persisted.

(The “Verify you are human by completing the action below.” / “Verify you are human” checkbox is also pretty false, as ticking the box in no way verifies you are human, but that feels slightly less disingenuous.)

Re: Cloudflare outage on November 18, 2025 post mortem

#545

Post by selfmodruntime »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

While this is true, I wish that Rust had more of a first-class support for `no_panic`. Every solution we do have is hacky. I wish that I could guarantee that there were no panic calls anywhere in a code path.

Re: Cloudflare outage on November 18, 2025 post mortem

#546

Post by gwd »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is the multi-million dollar .unwrap() story.

While there are certainly many things to admire about Rust, this is why I prefer Golang's "noisy" error handling. In golang that would be either:

    feature_values, err := features.append_with_names(...)
And the compiler would have complained that this value of `err` was unused; or you'd write:

    feature_values, _ := features.append_with_names(...)
And it would be far more obvious that an error message is being ignored.

(Renaming `unwrap` to `unwrapOrPanic` would probably help too.)

Re: Cloudflare outage on November 18, 2025 post mortem

#547

Post by selfmodruntime »

tormeh wrote: post #525
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code. And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.

I would love to go further and explicitely forbid unwrap and similar calls using a `no_panic` attribute.

Re: Cloudflare outage on November 18, 2025 post mortem

#548

Post by 9rx »

ironman1478 wrote: post #261
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I'm not a fan of rust, but I don't think that is the only takeaway. All systems have assumptions about their input and if the assumption is violated, it has to be caught somewhere. It seems like it was caught too deep in the system. Maybe the validation code should've handled the larger size, but also the db query produced something invalid. That shouldn't have ever happened in the first place.

The takeaway here isn’t about Rust itself, but that the Rust marketing crew’s claims that we constantly read on HN and elsewhere about the Result type magically saving you from making mistakes is not a good message to send.

Re: Cloudflare outage on November 18, 2025 post mortem

#549

Post by frumplestlatz »

brabel wrote: post #495
wrs wrote: post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality.

Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled.

If you need the value, you need to handle the case that it’s not available explicitly. You need to define your error path(s)

Anything else leads to, well, this.

Post reply on HN