Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

741–750 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#741

Post by miyuru »

aquariusDue wrote: post #706
polack wrote: post #653

Earlier quoted context omitted.

What is not realistic? To do simple input validation on data that has the potential to break 20% of the internet? To not have a system in place to rollback to the latest known state when things crash? Cloudflare builds a global scale system, not an iphone app. Please act like it.

Yeah, I don't quite understand the people cutting Cloudflare massive slack. It's not about nailing blame on a single person or a team, it's about keeping a company that is THE closest thing to a public utility for the web accountable. They more or less did a Press Release with a call to action to buy or use their services at the end and everybody is going "Yep, that's totally fine. Who hasn't sent a bug to prod, amir…

>It goes over my head why Cloudflare is HN's darling while others like Google, Microsoft and AWS don't usually enjoy the same treatment.

Do the others you mentioned provide such detailed outage reports, within 24 hours of an incident? I’ve never seen others share the actual code that related to the incident.

Or the CEO or CTO replying to comments here?

>Press Release

This is not press release, they always did these outage posts from the start of the company.

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

Re: Cloudflare outage on November 18, 2025 post mortem

#742

Post by quotemstr »

Veliladon wrote: post #726
quotemstr wrote: post #685

Earlier quoted context omitted.

If the error had been an exception instead of a result, could have bubbled up I have been saying for years that Rust botched error handling in unfixable ways. I will go to the grave believing Rust fumbled. The design of the Rust language encourages people to use unwrap() to turn foreseeable runtime problems into fatal errors. It's the path of least resistance, so people will take it. Rust encourages developers to con…

Errors work just like exceptions especially if you use the ? operator and let the error bubble up the chain. This is the Rust equivalent of an unhandled exception and the ripcord being pulled.

In C++, functions are error-colored by default. You write "noexcept" if you want your function to be infallible-colored instead.

(You usually want to make a function infallible if you're using your noexcept function as part of a cleanup path or part of a container interface that allows for more optimizations of it knows certain container operations are infallible.)

Rust makes infallibility the syntactic default and makes you write Result to indicate fallibility. People often don't want to color their functions this way. Guess what happens when a programmer is six levels deep in infallible-colored function calls and does something that can fail.

.unwrap()

Guess what, in Rust, is fallible?

Mutex acquire.

Guess what you need to do often on infallible cleanup paths?

Mutex acquire.

Re: Cloudflare outage on November 18, 2025 post mortem

#743

Post by evntdrvn »

ignoramous wrote: post #431
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

> Their bot management system is designed to push a configuration out to their entire network rapidly. Once every 5m is not "rapidly". It isn't uncommon for configuration systems to do it every few seconds [0]. > While it’s certainly useful to examine the root cause in the code. Believe the issue is as much an output from a periodic run (clickhouse query) caused by (on the surface, an unrelated change) causing this f…

Thanks for sharing that AWS doc

Re: Cloudflare outage on November 18, 2025 post mortem

#744

Post by throw0101c »

rkachowski wrote: post #579
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Classic @devops_borat "To make error is human. To propagate error to all server in automatic way is #devops"

> "To make error is human. To propagate error to all server in automatic way is #devops"

This saying dates back to 1969: To err is human but to really foul things up requires a computer.

* https://quoteinvestigator.com/2010/12/07/foul-computer/

Also: I know there’s a proverb which says ‘To err is human,’ but a human error is nothing to what a computer can do if it tries.

* https://quoteinvestigator.com/2017/05/26/computer-error/

Re: Cloudflare outage on November 18, 2025 post mortem

#745

Post by torginus »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

By the way - does this discussion matter and were they wrong to use unwrap()?

The way they wrote the code means that having more than 200 features is a hard non-transient error - even if they recovered from it, it meant they'd have had the same error when the code got to the same place.

I'm sure when the process crashed, k8s restarted the pod or something - then it reran the same piece of code and crashed in the same place.

While I don't necessarily agree with crashing as business strategy, I don't think that doing anything other than either dropping the extra rules or allocating more memory - neither of which the original code was built to do (probably by design).

The code made the local hard assumption that there won't ever be more than 200 rules and its okay to crash if that count is exceeded.

If you design your code around an invariant never being violated (which is fine), you have to make it clear on a higher level that they did.

This isn't a Rust problem (though Rust does make it easy to do the wrong thing here imo)

Re: Cloudflare outage on November 18, 2025 post mortem

#746

Post by throw0101c »

matternot wrote: post #576
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

I don't understand why they didn't validate and sanitize the new config file revision. If bad(whatever that reason is) throw an error and revert back to previous version. You don't need to take down the whole internet for that.

> I don't understand why they didn't validate and sanitize the new config file revision.

The new config file was not (AIUI) invalid (syntax-wise) but rather too big:

> […] That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.

> The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.

Re: Cloudflare outage on November 18, 2025 post mortem

#747

Post by underdeserver »

torginus wrote: post #738
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Say what you want exception haters, but at least in exceptions-as-default languages, the decision of a particular issues is fatal to the whole program can be decided centrally at a high level, and not every choice is forced to be up to individual discretion.

But you can do the same thing with Rust, by piping up Results.

Re: Cloudflare outage on November 18, 2025 post mortem

#748

Post by nip »

polack wrote: post #570
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

It’s easy to pick on logic that failed and for which you have a very detailed and great post mortem write-up.

Yet you omit to acknowledge that the remaining 99.99999% logic written that powers Cloudflare works flawlessly.

Also, hindsight is 20/20

Re: Cloudflare outage on November 18, 2025 post mortem

#750

Post by zf00002 »

As an IT person, I wonder what it's like to work for a company like this. Where presumably IT stuff has a priority. Unlike the companies I've worked for where IT takes a backseat to everything until something goes wrong. Company I work had a huge new office built, with the plan it would be big enough for future growth, yet despite repeated attempts to reserve a larger space, our server room and infrastructure is actually smaller than our old building and has no room to grow.
Post reply on HN