Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

561–570 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#561

Post by gwd »

brabel wrote: post #495
wrs wrote: post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

> Yes, I always thought it was wrong to use unwrap in examples.

And because it gets picked up by LLMs. It would be interesting to know if this particular .unwrap() was written by a human.

Re: Cloudflare outage on November 18, 2025 post mortem

#562

Post by watchful_moose »

pas wrote: post #473
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Rolling out new code should be done differently than rolling out new data to fight bots. If every time there's a new bot someone needs to write code that can blow up their whole service, maybe they need to iterate a bit on this design?

This isn't what they do, though. This is a data/config push - original article says _a “feature file” used by our Bot Management system_

Re: Cloudflare outage on November 18, 2025 post mortem

#564

Post by selfmodruntime »

karel-3d wrote: post #468
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

As a gopher I never understand why is there so many unwraps in an average rust code. Average Go code has much less panics than Rust has unwraps, which are functionally equivalent.

I love Go and write a ton of it. I've had real segfaults quite a lot.

Re: Cloudflare outage on November 18, 2025 post mortem

#565

Post by selfmodruntime »

arccy wrote: post #89
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

This is untrue. A `?` operator would have done just fine here. I agree with you though that it should be possible to explicitely forbid unwraps.

Re: Cloudflare outage on November 18, 2025 post mortem

#566

Post by yakshaving_jgt »

echelon wrote: post #346
wrs wrote: post #238

Earlier quoted context omitted.

I didn't restate the context from the code we're discussing: it must not panic. If you don't care if the code panics, then go ahead and unwrap/expect/index, because that conforms to your chosen error handling scheme. This is fine for lots of things like CLI tools or isolated subprocesses, and makes review a lot easier. So: first, identify code that cannot be allowed to panic. Within that code, yes, in the rare case t…

Cargo needs to grow a label for crates that provably do not panic. (Neverminding allocations and things outside our control flow.) I want to ban crates that panic from my dependency chain. The language could really use an extra set of static guarantees around this. I would opt in.

This sounds a little bit like Safe Haskell, which never really took off.

Re: Cloudflare outage on November 18, 2025 post mortem

#567

Post by abalone »

dev_l1x_be wrote: post #484
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Exactly the right take. Even when you want to have rapid changes on your infra, do it at least by region. You can start with the region where the least amount of users are impacted and if everything is fine, there is no elevated number of crashes for example, you can move forward. It was a standard practice at $RANDOM_FAANG when we had such deployments.

Thank you. I am sympathetic to CF’s need to deploy these configs globally fast and don’t think slowing down their DDoS mitigation is necessarily a good trade off. What I am saying is this presents a bigger reliability risk and needs correspondingly fine crafted observability around such config changes and a rollback runbook. Greater risk -> greater attention.

Re: Cloudflare outage on November 18, 2025 post mortem

#568

Post by yakshaving_jgt »

yoyohello13 wrote: post #101
arccy wrote: post #89

Earlier quoted context omitted.

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

So… basically every language ever? Except maybe Haskell.

It's easy to cause this kind of failure in Haskell also.

Re: Cloudflare outage on November 18, 2025 post mortem

#569

Post by gkoz »

Given this was triggered by an old school configuration change across multiple servers, there's too little discussion of that particular process.

It sounds like the change could've been rolled out more slowly, halted when the incident started and perhaps rolled back just in case.

Re: Cloudflare outage on November 18, 2025 post mortem

#570

Post by polack »

abalone wrote: post #380
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

They failed on so many levels here.

How can you write the proxy without handling the config containing more than the maximum features limit you set yourself?

How can the database export query not have a limit set if there is a hard limit on number of features?

Why do they do non-critical changes in production before testing in a stage environment?

Why did they think this was a cyberattack and only after two hours realize it was the config file?

Why are they that afraid of a botnet? Does not leave me confident that they will handle the next Aisuru attack.

I'm migrating my customers off Cloudflare. I don't think they can swallow the next botnet attacks and everyone on Cloudflare go down with the ship, so it will be safer to not be behind Cloudflare when it hits.

Post reply on HN