Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

431–440 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#431

Post by ignoramous »

abalone wrote: post #380
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

> Their bot management system is designed to push a configuration out to their entire network rapidly.

Once every 5m is not "rapidly". It isn't uncommon for configuration systems to do it every few seconds [0].

> While it’s certainly useful to examine the root cause in the code.

Believe the issue is as much an output from a periodic run (clickhouse query) caused by (on the surface, an unrelated change) causing this failure. That is, the system that validated the configuration (FL2) was different to the one that generated it (ML Bot Management DB).

Ideally, it is the system that vends a complex configuration that also vends & tests the library to consume it, or the system that consumes it, does so as if it was "tasting" the configuration first before devouring it unconditionally [1].

Of course, as with all distributed system failures, this is all easier said and done in hindsight.

[0] Avoiding overload in distributed systems by putting the smaller service in control (pg 4), https://d1.awsstatic.com/builderslibrary/pdfs/Avoiding%20ove...

[1] Lessons from CloudFront (2016), https://youtube.com/watch?v=n8qQGLJeUYA&t=1050

Re: Cloudflare outage on November 18, 2025 post mortem

#433

Post by tucnak »

gishh wrote: post #61
orphea wrote: post #40

Earlier quoted context omitted.

Like goto, unwrap is just a tool that has its use cases. No need to make a boogeyman out of it.

To be fair, if you’re not “this tall” you really shouldn’t consider using goto in a c program. Most people aren’t that tall.

Nonsense. Linux kernel for one example, uses goto everywhere for error handling.

Re: Cloudflare outage on November 18, 2025 post mortem

#434

Post by Ekaros »

HumanOstrich wrote: post #119
otterley wrote: post #112

Earlier quoted context omitted.

In my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference. People think of configuration updates (or state updates, call them what you will) as inherently safer than code updates, but history (and today!) demonstrates that they are not. Yet even experienced engineers will allow changes like these into production unattended -- even ones who wouldn't dare let a single…

They narrowed down the actual problem to some Rust code in the Bot Management system that enforced a hard limit on the number of configuration items by returning an error, but the caller was just blindly unwrapping it.

Sounds like lack of good testing. Too many items in any input should be a boundary case you will get to eventually.

Re: Cloudflare outage on November 18, 2025 post mortem

#435

Post by jbs789 »

abalone wrote: post #380
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Thanks for this assessment.

In a productive way, this view also shifts the focus to improving the system (visibility etc), empowering the team, rather than focusing on the code which broke (probably strikes fear in the individuals, to do anything!)

Re: Cloudflare outage on November 18, 2025 post mortem

#436

Post by pdimitar »

lenkite wrote: post #414
pdimitar wrote: post #179

Earlier quoted context omitted.

There are even lints for this but people get impatient and just override them or fight for them to no longer be the default. As usual: people problem, not a tech problem. In the last years a lot of strides have been made. But people will be people.

Linting is not good enough. The compiler should refuse to compile the code without it marked with an explicit annotation. Too much Rust code is panic happy since using casual use of `unwrap` is perma-etched into everyone's minds by the amount of demo code out there using unwrap.

I completely agree. But IMO Rust would have not gained traction if it was as strict. It would be branded as an academic toy language.

But now after we are past that and it has a lot of mind share, I'd say it's time to start tightening the bolts.

Re: Cloudflare outage on November 18, 2025 post mortem

#437

Post by Hamuko »

ignoramous wrote: post #431
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

> Their bot management system is designed to push a configuration out to their entire network rapidly. Once every 5m is not "rapidly". It isn't uncommon for configuration systems to do it every few seconds [0]. > While it’s certainly useful to examine the root cause in the code. Believe the issue is as much an output from a periodic run (clickhouse query) caused by (on the surface, an unrelated change) causing this f…

>Once every 5m is not "rapidly".

Isn't rapidly more of how long it takes to get from A to Z rather than how often it is performed? You can push out a configuration update every fortnight but if it goes through all of your global servers in three seconds, I'd call it quite rapid.

Re: Cloudflare outage on November 18, 2025 post mortem

#438

Post by oskarkk »

keiywuvfwofw wrote: post #399

This post was written by chatgpt?? https://blog.cloudflare.com/18-november-2025-outage/#:~:text...

Here's a random post from their blog by the same author from 2017 with an em dash:

> As we wrote before, we believe Blackbird Tech's dangerous new model of patent trolling — where they buy patents and then act their own attorneys in cases — may be a violation of the rules of professional ethics.

https://blog.cloudflare.com/patent-troll-battle-update-doubl...

ChatGPT didn't invent the em dash, some people were always using it. But yeah, it's often one of the signs of AI.

Re: Cloudflare outage on November 18, 2025 post mortem

#439

Post by throwaway38294 »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

This is a bummer. The unwrap()'ing function already returned a result and should have just propagated the error. Presumably the caller could have handled more sensibly than just panic'ing.

Re: Cloudflare outage on November 18, 2025 post mortem

#440

Post by abalone »

ignoramous wrote: post #431
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

> Their bot management system is designed to push a configuration out to their entire network rapidly. Once every 5m is not "rapidly". It isn't uncommon for configuration systems to do it every few seconds [0]. > While it’s certainly useful to examine the root cause in the code. Believe the issue is as much an output from a periodic run (clickhouse query) caused by (on the surface, an unrelated change) causing this f…

By rapid I mean a rapid rollout of changes to 100% of the fleet, not how often changes are made.
Post reply on HN