Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

171–180 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#171

"Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero." This simply means, the exception handling quality of your new FL2 is non-existent and is not at par / code logic wise similar to FL. I hope it was not because of AI driven efficiency gains.

In most domains, silently returning 0 in a case where your logic didn't actually calculate the thing you were trying to calculate is far worse than giving a clear error.

Re: Cloudflare outage on November 18, 2025 post mortem

#172

Earlier quoted context omitted.

> Returning a Result by definition means the method can fail. No more than returning an int by definition means the method can return -2.

What? Results have a limited number of possible error states that are well defined.

Some call points to a function that returns a Result will never return an Error.

Some call points to a function that returns an int will never return -2.

Sometimes you know things the type system does not know.

Re: Cloudflare outage on November 18, 2025 post mortem

#173
While it's certainly worthwhile to discuss the Technical and Procedural elements that contributed to this Service Outage, the far more important (and mutually-exclusive aspect) to discuss should be:

Why have we built / permitted the building of / Subscribed to such a Failure-intolerant "Network"?

Re: Cloudflare outage on November 18, 2025 post mortem

#174
post #57

Earlier quoted context omitted.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…

You call this transparency, but fail to answer the most important questions: what was in the burrito? Was it good? Would you recommend?

Re: Cloudflare outage on November 18, 2025 post mortem

#175
post #2

> a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system ... to keep [that] system up to date with ever changing threats > The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail A configuration error can cause internet-scale outages. What an era…

The speed and transparency of Cloudflare publishing this port mortem is excellent.

I also found the "remediation and follow up" section a bit lacking, not mentioning how, in general, regressions in query results caused by DB changes could be caught in future before they get widely rolled out.

Even if a staging env didn't have a production-like volume of data to trigger the same failure mode of a bot management system crash, there's also an opportunity to detect that something has gone awry if there were tests that the queries were returning functionally equivalent results after the proposed permission change. A dummy dataset containing a single http_requests_features column would suffice to trigger the dupe results behaviour.

In theory there's a few general ways this kind of issue could be detected, e.g. someone or something doing a before/after comparison to test that the DB permission change did not regress query results for common DB queries, for changes that are expected to not cause functional changes in behaviour.

Maybe it could have been detected with an automated test suite of the form "spin up a new DB, populate it with some curated toy dataset, then run a suite of important queries we must support and check the results are still equivalent (after normalising row order etc) to known good golden outputs". This style of regression testing is brittle, burdensome to maintain and error prone when you need to make functional changes and update what then "golden" outputs are - but it can give a pretty high probability of detecting that a DB change has caused unplanned functional regressions in query output, and you can find out about this in a dev environment or CI before a proposed DB change goes anywhere near production.

Re: Cloudflare outage on November 18, 2025 post mortem

#176
post #143

Earlier quoted context omitted.

At Facebook they name certain "escape hatch" functions in a way that inescapably make them look like a GIANT EYESORE. Stuff like DANGEROUSLY_CAST_THIS_TO_THAT, or INVOKE_SUPER_EXPENSIVE_ACTION_SEE_YOU_ON_CODE_REVIEW. This really drives home the point that such things must not be used except in rare extraordinary cases. If unwrap() were named UNWRAP_OR_PANIC(), it would be used much less glibly. Even more, I wish ther…

right and if the language designers named it UNWRAP_OR_PANIC() then people would rightfully be asking why on earth we can't just use a try-catch around code and have an easier life

But a panic can be caught and handled safely (e.g. via std:: panic tools). I'd say that this is the correct use case for exceptions (ask Martin Fowler, of all people).

There is already a try/catch around that code, which produces the Result type, which you can presumptuously .unwrap() without checking if it contains an error.

Instead, one should use the question mark operator, that immediately returns the error from the current function if a Result is an error. This is exactly similar to rethrowing an exception, but only requires typing one character, the "?".

Re: Cloudflare outage on November 18, 2025 post mortem

#178

Earlier quoted context omitted.

right and if the language designers named it UNWRAP_OR_PANIC() then people would rightfully be asking why on earth we can't just use a try-catch around code and have an easier life

Probably not, since errors as values are way better than exceptions.

How so? An exception is a value that's given the closest, conceptually appropriate, point that was decided to handle the value, allowing you to keep your "happy path" as clean code, and your "exceptional circumstances" path at the level of abstraction that makes sense.

It's way less book-keeping with exceptions, since you, intentionally, don't have to write code for that exceptional behavior, except where it makes sense to. The return by value method, necessarily, implements the same behavior, where handling is bubbled up to the conceptually appropriate place, through returns, but with much more typing involved. Care is required for either, since not properly bubbling up an exception can happen in either case (no re-raise for exceptions, no return after handling for return).

Re: Cloudflare outage on November 18, 2025 post mortem

#179
post #122

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Some languages and style guides simply forbid throwing exceptions without catching / proper recovery. Google C++ bans exceptions and the main mechanism for propogating errors is `absl::Status` which the caller has to check. Not familiar with Rust but it seems unwrap is such a thing that would be banned.

There are even lints for this but people get impatient and just override them or fight for them to no longer be the default.

As usual: people problem, not a tech problem. In the last years a lot of strides have been made. But people will be people.

Re: Cloudflare outage on November 18, 2025 post mortem

#180

Earlier quoted context omitted.

It's great that you're working on regionalization. Yes, it is hard, but 100x harder if you don't start with cellular design in mind. And as I said in the root of the thread, this is a sign that CloudFlare needs to invest in it just like you have been.

I recoil from that last statement not because I have a rooting interest in Cloudflare but because the last several years of working at Fly.io have drilled Richard Cook's "How Complex Systems Fail"† deep into my brain, and what you said runs aground of Cook #18: Failure free operations require experience with failure. If the exact same thing happens again at Cloudflare, they'll be fair game. But right now I feel peopl…

Suppose they did have the cellular architecture today, but every other fact was identical. They'd still have suffered the failure! But it would have been contained, and the damage would have been far less.

Fires happen every day. Smoke alarms go off, firefighters get called in, incident response is exercised, and lessons from the situation are learned (with resulting updates to the fire and building codes).

Yet even though this happens, entire cities almost never burn down anymore. And we want to keep it that way.

As Cook points out, "Safety is a characteristic of systems and not of their components."

Post reply on HN