Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

71–80 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#71

Post by thesh4d0w »

yen223 wrote: post #57
SerCe wrote: post #9

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

The person who posted both this blog article and the hacker news post, is Matthew Prince, one of highly technical billionaire founders of cloudflare. I'm sure if he wants something to happen, it happens.

Re: Cloudflare outage on November 18, 2025 post mortem

#72

Post by tclancy »

hn-idiots wrote: post #65
bayesnet wrote: post #11

Earlier quoted context omitted.

And a well-written one at that. Compared to the AWS port-mortem this could be literature.

[flagged]

I feel like your username really brings something extra to the party. Now go home.

Re: Cloudflare outage on November 18, 2025 post mortem

#73

Post by nikcub »

otterley wrote: post #19

> work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditio…

They require the bot management config to update and propagate quickly in order to respond to attacks - but this seems like a case where updating a since instance first would have seen the panic and stopped the deploy.

I wonder why clickhouse is used to store the feature flags here, as it has it's own duplication footguns[0] which could have also easily lead to a query blowing up 2/3x in size. oltp/sqlite seems more suited, but i'm sure they have their reasons

[0] https://clickhouse.com/docs/guides/developer/deduplication

Re: Cloudflare outage on November 18, 2025 post mortem

#74

Post by mewpmewp2 »

otterley wrote: post #19

> work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditio…

It seems they had this continous rollout for the config service, but the services consuming this were affected even by small percentage of these config providers being faulty, since they were auto updating every few minutes their configs. And it seems there is a reason for these updating so fast, presumably having to react to threat actors quickly.

Re: Cloudflare outage on November 18, 2025 post mortem

#76

Post by tom1337 »

yen223 wrote: post #57
SerCe wrote: post #9

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

I mean the CEO posted the post-mortem so there aren't that many layers of stakeholders above. For other post-mortems by engineers, Matthew once said that the engineering team is running the blog and that he wouldn't event know how to veto even if he wanted [0]

[0] https://news.ycombinator.com/item?id=45588305

Re: Cloudflare outage on November 18, 2025 post mortem

#77

Post by jryio »

dzonga wrote: post #54

> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually…

You misunderstand what Rust’s guarantees are. Rust has never promised to solve or protect programmers from logical or poor programming. In fact, no such language can do that, not even Haskell.

Unwrapping is a very powerful and important assertion to make in Rust whereby the programmer explicitly states that the value within will not be an error, otherwise panic. This is a contract between the author and the runtime. As you mentioned, this is a human failure, not a language failure.

Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought… (n.b. nginx)

This is the classic example of when something fails, the failure cause over indexes on - while under indexing on the quadrillions of memory accesses that went off without a single hitch thanks to the borrow checker.

I postulate that whatever the cost in millions or hundreds of millions of dollars by this Cloudflare outage, it has paid for more than by the savings of safe memory access.

See: https://en.wikipedia.org/wiki/Survivorship_bias

Re: Cloudflare outage on November 18, 2025 post mortem

#78

Post by thundergolfer »

tptacek wrote: post #21
tristan-morris wrote: post #16

Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.

Probably because this case was something more akin to an assert than an error check.

Fly writes a lot of Rust, do you allow `unwrap()` in your production environment? At Modal we only allow `expect("...")` and the message should follow the recommended message style[1].

I'm pretty surprised that Cloudflare let an unwrap into prod that caused their worst outage in 6 years.

1. https://doc.rust-lang.org/std/option/enum.Option.html#recomm...

Re: Cloudflare outage on November 18, 2025 post mortem

#79

Post by eastdakota »

tclancy wrote: post #72
hn-idiots wrote: post #65

Earlier quoted context omitted.

[flagged]

I feel like your username really brings something extra to the party. Now go home.

Can attest: not a single LLM used. Couldn’t if I tried. Old school. And not entirely proud of that.

Re: Cloudflare outage on November 18, 2025 post mortem

#80

Post by otterley »

mewpmewp2 wrote: post #74
otterley wrote: post #19

> work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditio…

It seems they had this continous rollout for the config service, but the services consuming this were affected even by small percentage of these config providers being faulty, since they were auto updating every few minutes their configs. And it seems there is a reason for these updating so fast, presumably having to react to threat actors quickly.

It's in everyone's interest to mitigate threats as quickly as possible. But it's of even greater interest that a core global network infrastructure service provider not DOS a significant proportion of the Internet by propagating a bad configuration too quickly. The key here is to balance responsiveness against safety, and I'm not sure they struck the right balance here. I'm just glad that the impact wasn't as long and as severe as it could have been.
Post reply on HN