Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#111

Post by SchwKatze »

arccy wrote: post #89
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

Unwrap isn't a synonym for laziness, it's just like an assertion, when you do unwrap() you're saying the Result should NEVER fail, and if does, it should abort the whole process. What was wrong was the developer assumption, not the use of unwrap.

Re: Cloudflare outage on November 18, 2025 post mortem

#112

Post by otterley »

tptacek wrote: post #98
otterley wrote: post #80

Earlier quoted context omitted.

It's in everyone's interest to mitigate threats as quickly as possible. But it's of even greater interest that a core global network infrastructure service provider not DOS a significant proportion of the Internet by propagating a bad configuration too quickly. The key here is to balance responsiveness against safety, and I'm not sure they struck the right balance here. I'm just glad that the impact wasn't as long an…

This isn't really "configuration" so much as it is "durable state" within the context of this system.

In my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference.

People think of configuration updates (or state updates, call them what you will) as inherently safer than code updates, but history (and today!) demonstrates that they are not. Yet even experienced engineers will allow changes like these into production unattended -- even ones who wouldn't dare let a single line of code go live without being subject to the full CI/CD process.

Re: Cloudflare outage on November 18, 2025 post mortem

#113

Post by HumanOstrich »

nikcub wrote: post #73
otterley wrote: post #19

> work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditio…

They require the bot management config to update and propagate quickly in order to respond to attacks - but this seems like a case where updating a since instance first would have seen the panic and stopped the deploy. I wonder why clickhouse is used to store the feature flags here, as it has it's own duplication footguns[0] which could have also easily lead to a query blowing up 2/3x in size. oltp/sqlite seems more…

I don't think sqlite would come close to their requirements for permissions or resilience, to name a couple. It's not the solution for every database issue.

Also, the link you provided is for eventual deduplication at the storage layer, not deduplication at query time.

Re: Cloudflare outage on November 18, 2025 post mortem

#114

Post by anurag »

eastdakota wrote: post #95
yen223 wrote: post #57

Earlier quoted context omitted.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…

Appreciate the extra transparency on the process.

Re: Cloudflare outage on November 18, 2025 post mortem

#115

Post by fwjafwasd »

orphea wrote: post #40
sayrer wrote: post #18

Earlier quoted context omitted.

Yes, can't have .unwrap() in production code (it's ok in tests)

Like goto, unwrap is just a tool that has its use cases. No need to make a boogeyman out of it.

panicans should be using .expect() in production

Re: Cloudflare outage on November 18, 2025 post mortem

#116

Post by yoyohello13 »

People really like to hate on Rust for some reason. This wasn’t a Rust problem, no language would have saved them from this kind of issue. In fact, the compiler would have warned that this was a possible issue.

I get it, don’t pick languages just because they are trendy, but if any company’s use case is a perfect fit for Rust it’s cloudflare.

Re: Cloudflare outage on November 18, 2025 post mortem

#117

Post by JamesJGoodwin »

>Currently that limit is set to 200, well above our current use of ~60 features. Again, the limit exists because for performance reasons we preallocate memory for the features.

So they basically hardcoded something, didn't bother to cover the overflow case with unit tests, didn't have basic error catching that would fallback and send logs/alerts to their internal monitoring system and this is why half of the internet went down?

Re: Cloudflare outage on November 18, 2025 post mortem

#118

Post by testemailfordg2 »

"Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero."

This simply means, the exception handling quality of your new FL2 is non-existent and is not at par / code logic wise similar to FL.

I hope it was not because of AI driven efficiency gains.

Re: Cloudflare outage on November 18, 2025 post mortem

#119

Post by HumanOstrich »

otterley wrote: post #112
tptacek wrote: post #98

Earlier quoted context omitted.

This isn't really "configuration" so much as it is "durable state" within the context of this system.

In my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference. People think of configuration updates (or state updates, call them what you will) as inherently safer than code updates, but history (and today!) demonstrates that they are not. Yet even experienced engineers will allow changes like these into production unattended -- even ones who wouldn't dare let a single…

They narrowed down the actual problem to some Rust code in the Bot Management system that enforced a hard limit on the number of configuration items by returning an error, but the caller was just blindly unwrapping it.

Re: Cloudflare outage on November 18, 2025 post mortem

#120

Post by yahoozoo »

eastdakota wrote: post #79
tclancy wrote: post #72

Earlier quoted context omitted.

I feel like your username really brings something extra to the party. Now go home.

Can attest: not a single LLM used. Couldn’t if I tried. Old school. And not entirely proud of that.

Based CEO
Post reply on HN