Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

131–140 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#131

Post by SchemaLoad »

SchwKatze wrote: post #111
arccy wrote: post #89

Earlier quoted context omitted.

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

Unwrap isn't a synonym for laziness, it's just like an assertion, when you do unwrap() you're saying the Result should NEVER fail, and if does, it should abort the whole process. What was wrong was the developer assumption, not the use of unwrap.

It also makes it very obvious in the code, something very dangerous is happening here. As a code reviewer you should see an unwrap() and have alarm bells going off. While in other languages, critical errors are a lot more hidden.

Re: Cloudflare outage on November 18, 2025 post mortem

#132

Post by Klonoar »

dzonga wrote: post #54

> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually…

> I don't use Rust, but a lot of Rust people say if it compiles it runs.

Do you grok what the issue was with the unwrap, though...?

Idiomatic Rust code does not use that. The fact that it's allowed in a codebase says more about the engineering practices of that particular project/module/whatever. Whoever put the `unwrap` call there had to contend with the notion that it could panic and they still chose to do it.

It's a programmer error, but Rust at least forces you to recognize "okay, I'm going to be an idiot here". There is real value in that.

Re: Cloudflare outage on November 18, 2025 post mortem

#134

Post by tptacek »

otterley wrote: post #125
HumanOstrich wrote: post #119

Earlier quoted context omitted.

They narrowed down the actual problem to some Rust code in the Bot Management system that enforced a hard limit on the number of configuration items by returning an error, but the caller was just blindly unwrapping it.

A dormant bug in the code is usually a condition precedent to incidents like these. Later, when a bad input is given, the bug then surfaces. The bug could have laid dormant for years or decades, if it ever surfaced at all. The point here remains: consider every change to involve risk, and architect defensively.

They made the classic distributed systems mistake and actually did something. Never leap to thing-doing!

Re: Cloudflare outage on November 18, 2025 post mortem

#135

Post by JuniperMesos »

arccy wrote: post #89
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

It's a little subtler than this. You want it to be easy to not handle an error while developing, so you can focus on getting the core logic correct before error-handling; but you want it to be hard to deploy or release the software without fully handling these checks. Some kind of debug vs release mode with different lints seems like a reasonable approach.

Re: Cloudflare outage on November 18, 2025 post mortem

#136

Post by leshenka »

arccy wrote: post #89
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

All languages with few exceptions have these kinds of escape hatches like unwrap

Re: Cloudflare outage on November 18, 2025 post mortem

#137

Post by SchemaLoad »

lmm wrote: post #128
dzonga wrote: post #54

> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually…

> Rust won't save you from the usual programming mistake. Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like. No language will prevent you from writing broken code if you're determined to do so, and no language is impossible to write correct code in if you…

Yep, unwrap() and unsafe are escape hatches that need very good justifications. It's fine for casual scripts where you don't care if it crashes. For serious production software they should be either banned, or require immense scrutiny.

Re: Cloudflare outage on November 18, 2025 post mortem

#138

Post by wildmXranat »

Hold up ,- when I used a C or similar language for accessing a database and wanted to clamp down on memory usage to deterministically control how much I want to allocated, I would explicitly limit the number of rows in the query.

There never was an unbound "select all rows from some table" without a "fetch first N rows only" or "limit N"

If you knew that this design is rigid, why not leverage the query to actually do it ?

What am I missing ?

Re: Cloudflare outage on November 18, 2025 post mortem

#139

Post by tptacek »

otterley wrote: post #112
tptacek wrote: post #98

Earlier quoted context omitted.

This isn't really "configuration" so much as it is "durable state" within the context of this system.

In my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference. People think of configuration updates (or state updates, call them what you will) as inherently safer than code updates, but history (and today!) demonstrates that they are not. Yet even experienced engineers will allow changes like these into production unattended -- even ones who wouldn't dare let a single…

Reframe this problem: instead of bot rules being propagated, it's the enrollment of a new customer or a service at an existing customer --- something that must happen at Cloudflare several times a second. Does it still make sense to you to think about that in terms of "pushing new configuration to prod"?

Re: Cloudflare outage on November 18, 2025 post mortem

#140

Post by zmj »

koakuma-chan wrote: post #38
tristan-morris wrote: post #16

Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.

Why is there a 200 limit on appending names?

Everything has a limit. You can define it, or be surprised when you find out what it is.
Post reply on HN