Cloudflare outage on November 18, 2025 post mortem • Hacker News Board

Re: Cloudflare outage on November 18, 2025 post mortem

#641

Post by BrtByte » Wed, Nov 19, 2025, 11:42 AM UTC

abalone wrote: post #380

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

You can write the safest code in the world, but if you're shipping config changes globally every few minutes without a robust rollback plan or telemetry that pinpoints when things go sideways, you're flying blind

Re: Cloudflare outage on November 18, 2025 post mortem

#642

Post by polack » Wed, Nov 19, 2025, 11:43 AM UTC

jve wrote: post #582

polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> I'm migrating my customers off Cloudflare. Is that an overreaction? Name me global, redundant systems that have not (yet) failed. And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic. I mean no service have 100% uptime - just that some have more nines than others.

Yes, it's probably an overreaction.

But at the same time, what value do they add if they:

* Took down the the customers sites due to their bug.

* Never protected against an attack that our infra could not have handled by itself.

* Don't think that they will be able to handle the "next big ddos" attack.

It's just an extra layer of complexity for us. I'm sure there are attacks that could help our customers with, that's why we're using them in the first place. But until the customers are hit with multiple ddos attacks that we can not handle ourself then it's just not worth it.

Re: Cloudflare outage on November 18, 2025 post mortem

#643

Post by kqr » Wed, Nov 19, 2025, 11:45 AM UTC

One of the remediations listed is

> Eliminating the ability for core dumps or other error reports to overwhelm system resources

but this is not mentioned at all in the timeline above. My best guess would be that the process got stuck in a tight restart loop and filled available disk space with logs, but I'm happy to hear other guesses for people more familiar with Rust.

Re: Cloudflare outage on November 18, 2025 post mortem

#644

Post by mlrtime » Wed, Nov 19, 2025, 11:46 AM UTC

abalone wrote: post #380

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

I've also led a team of Incident Commanders at a FAANG.

If this was a routine config change, I could see how it could take 2 hours to start the mediation plan. However they should have dashboards that correlate config setting changes with 500 errors (or equivalent). It gets difficult when you have many of of these going out at the same time and they are slowly rolled out.

The root cause document is mostly for high level and the public. The details on this specific outage will be in a internal document with many action items, some of them maybe quarter long projects including fixing this specific bug and maybe some linter/monitor to prevent it from happening again.

Re: Cloudflare outage on November 18, 2025 post mortem

#645

Post by BrtByte » Wed, Nov 19, 2025, 11:48 AM UTC

SerCe wrote: post #9

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

It's not just a PR-friendly summary either... they included real technical detail, timestamps, even code snippets

Re: Cloudflare outage on November 18, 2025 post mortem

#646

Post by echelon » Wed, Nov 19, 2025, 11:50 AM UTC

frumplestlatz wrote: post #549

brabel wrote: post #495

Earlier quoted context omitted.

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality. Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled. If you need the value, you need to handle the case that it’s not available explicitly. Y…

This.

This is a failure caused by lazy Rust programming and not relying on the language's design features.

It's a shame this code can even be written. It is surprising and escapes the expected safety of the language.

I'm terrified of some dependency using unwrap() or expect() and crashing for something entirely outside of my control.

We should have an opt-in strict Cargo.toml declaration that forbids compilation of any crate that uses entirely preventable panics. The only panics I'll accept are those relating to memory allocation.

This is one of the sharpest edges in the language, and it needs to be smoothed away.

Re: Cloudflare outage on November 18, 2025 post mortem

#647

Post by BrtByte » Wed, Nov 19, 2025, 11:51 AM UTC

yen223 wrote: post #57

SerCe wrote: post #9

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

Cloudflare seems to have baked this level of transparency into their culture and incident response process

Re: Cloudflare outage on November 18, 2025 post mortem

#648

Post by littlestymaar » Wed, Nov 19, 2025, 11:53 AM UTC

wrs wrote: post #418

littlestymaar wrote: post #402

Earlier quoted context omitted.

> In production code an unwrap or expect should be reviewed exactly like a panic. An unwrap should never make it to production IMHO. It's fine while prototyping, but once the project gets closer to production it's necessary to just grep `uncheck` in your code and replace those that can happen with a proper error management and replace those that cannot happen with `expect`, with a clear justification of why they cann…

I would say, sure, if you feel the same way about panic calls making to production. In other words, review all of them the same way. Because writing unwrap/expect is exactly the same as writing “if error, panic”.

I don't understand your point: panic! is akin to expect: you think about it consciously, use it explicitly and you write down a panic message explaining its rational.

unwrap isn't like that.

Re: Cloudflare outage on November 18, 2025 post mortem

#649

Post by mewpmewp2 » Wed, Nov 19, 2025, 11:53 AM UTC

Aeolun wrote: post #90

mewpmewp2 wrote: post #62

Earlier quoted context omitted.

But now consider how much extra data Cloudflare at its size would have to have just for staging, doubling or more their costs to have stage exactly as production. They would have to simulate similar amount of requests on top of themselves constantly since presumably they have 100s or 1000s of deployments per day. In this case it seems the database table in question seemed modest in size (the features for ML) so naive…

That just means it takes longer to test. It may not be possible to do it in a reasonable timeframe with the volumes involved, but if you already have 100k servers running to serve 25M requests per second, maybe briefly booting up another 100k isn’t going to be the end of the world? Either way, you don’t need to do it on every commit, just often enough that you catch these kinds of issues before they go to prod.

But they are probably doing hundreds of deployments a day, so that would make their pipelines extremely long? Not to mention costs.

Re: Cloudflare outage on November 18, 2025 post mortem

#650

Post by keepamovin » Wed, Nov 19, 2025, 11:54 AM UTC

That's interesting. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. It's like the issues with HOSTS.TXT needing to be copied among the network of the early internet to allow routing (taking days to download etc) and DNS having to be created to make that propagation less unwieldy.