Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

591–600 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#591

The most surprising thing to me here is that it took 3 hours to root cause, and points to a glaring hole in the platform observability. Even taking into account the fact that the service was failing intermittently at first, it still took 1.5 hours after it started failing consistently to root cause. But the service was crashing on startup. If a core service is throwing a panic at startup like that, it should be raisi…

If one actually looks at the current pingora API, it has limited ability to initialize async components at startup - the current pattern seems to be to lazily initialize on first call. An obvious downside of this is that a service can startup in a broken state. e.g. https://github.com/cloudflare/pingora/issues/169

I can imagine that this could easily lead to less visibility into issues.

Re: Cloudflare outage on November 18, 2025 post mortem

#592

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

I don't understand why they didn't validate and sanitize the new config file revision. If bad(whatever that reason is) throw an error and revert back to previous version. You don't need to take down the whole internet for that.

Same as for almost every bug I think: the dev in question hadn't considered that the input could be bad in the way that it turned out to be. Maybe they were new, or maybe they hadn't slept much because of a newborn baby, or maybe they thought it was a reasonable assumption that there would never be more than 200 ML features in the array in question. I don't think this developer will ever make the same mistake again at least.

Let those who have never written a bug before cast the first stone.

Re: Cloudflare outage on November 18, 2025 post mortem

#593

Earlier quoted context omitted.

> Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like. No one treats it like that and nearly every Rust project is filled with unwraps all over the place even in production system like Cloudflare's.

Well let me avoid those that don’t understand it. It’s literally Rust 101.

It's literally not, Rust tutorials are littered with `.unwrap()` calls. It might be Rust 102, but the first impression given is that the language is surprisingly happy with it.

Re: Cloudflare outage on November 18, 2025 post mortem

#594
post #570

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> Why did they think this was a cyberattack and only after two hours realize it was the config file?

They explain that at some length in TFA.

Re: Cloudflare outage on November 18, 2025 post mortem

#595
post #582

Earlier quoted context omitted.

> I'm migrating my customers off Cloudflare. Is that an overreaction? Name me global, redundant systems that have not (yet) failed. And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic. I mean no service have 100% uptime - just that some have more nines than others.

There are many self-hosted alternatives to protect against botnet. We don't have to use cloudflare. Everthing is under their control!

> There are many self-hosted alternatives to protect against botnet

Whatever you do, unless you have their bandwidth capacity, at some point those "self-hosted" will get flooded with traffic.

Re: Cloudflare outage on November 18, 2025 post mortem

#596

Earlier quoted context omitted.

Alternatively you can look at actually innovative programming languages to peek at the next 20 years of innovation. I am not sure that watching the trendy forefront successfully reach the 1990s and discuss how unwrapping Option is potentially dangerous really warm my heart. I can’t wait for the complete meltdown when they discover effect systems in 2040. To be more serious, this kind of incident is yet another remind…

Let's try to make effect systems happen quicker than that. > I mean this Cloudfare outage probably cost millions of dollars of damage in aggregate between lost revenue and lost productivity. How much of that will they actually have to pay? Probably nothing, because most paying customers of cloudflare are probably signing away their rights to sue Cloudflare for damages by being down for a while when they purchase Clou…

> I honestly have a hard time suggesting that those customers are individually wrong to do so - Cloudflare isn't down that often, and whatever amount it cost any individual customer by being down today might be more than offset by the DDOS protection they're buying.

We have collectively agreed to a world where software service providers have no incentive to be reliable as they are shielded from the consequences of their mistakes and somehow we see it as acceptable that software have a ton of issues and defects. The side effect is that research on actually lowering the cost of safety has little return on investment. It doesn't have be so.

> Anyway if you want Cloudflare regulated to prevent this, name the specific regulations you want to see.

I want software provider to be liable for the damage they cause and minimum quality regulation on par with an actual engineering discipline. I have always been astounded that nearly all software licences start with extremely broad limitation of liability provisions and people somehow feel fine with it. Try to extend that to any other product you regularly use in your life and see how that makes you fell.

How to do proper testing, formal methods and resilient design have been known for decades. I would personnaly be more than okay with let's move less fast and stop breaking things.

Re: Cloudflare outage on November 18, 2025 post mortem

#597

Earlier quoted context omitted.

This tweet thread invokes genuine despair in me. Do we really have to outsource even our tweets to LLMs? Really? I mean, I get spambots and the like tweeting mass-produced slop. But what compels a former engineer of the company in question to offer LLM-generated "insight" to the outage? Why? For what purpose? * For clarity, I am aware that the original tweets are written in Chinese, and they still have the stench of…

Out of interest... apart from the em dash, how else can you tell it's an LLM response? What are the telltales signs?

This particular excerpt is reeking of it with pretty much every line. I'll point out the patterns in the English translation, but all of these patterns apply cross-language.

> classic chain reaction triggered by “hidden assumptions + configuration chains”

"Classic/typical "x + y"", particularly when diagnosing an issue. This one is a really easy tell because humans, on aggregate, do not use quotation marks like this. There is absolutely no reason to quote these words here, and yet LLMs will do a combined quoted "x + y" where a human would simply write something natural like "hidden assumptions and configuration chains" without extraneous quotes.

> The configuration system requires “bad → reject, keep last-known-good” logic.

Another pattern with overeager usage of quotes is this ""x → y, z"" construct with very terse wording.

> This wasn't an attack, but a classic chain reaction

LLMs aggressively use "Not X, but Y". This is also a construct commonly used by humans, of course, but aside from often being paired with an em-dash, another tell is whether it actually contributes anything to the sentence. "Not X, but Y" is strongly contrasting and can add a dramatic flair to the thing being constrasted, but LLMs overuse it on things that really really don't need to be dramatised or contrasted.

> Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.

Two lists of three concepts back-to-back. LLMs enjoy, love, and adore this construct.

> Hats off to Cloudflare's engineers—those on the front lines putting out fires bear the brunt of such incidents.

This kind of completely vapid, feel-good word soup utilising a heroic analogy for something relatively mundane is another tell.

And more broadly speaking, there's a sort of verbosity and emptiness of actual meaning that permeates through most LLM writing. This reads absolutely nothing like what an engineer breaking down an outage looks like. Like, the aforementioned line of... "Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.". What is that actually communicating to you? It piles on technical lingo and high-level concepts in a way that is grammatically correct but contains no useful information for the reader.

Bad writing exists, of course. There's plenty of bad writing out there on the internet, and some of it will suffer from flaws like these even when written by a human, and some humans do like their em-dashes. But it's generally pretty obvious when the writing is taken on aggregate and you see recognisable pattern after pattern combined with em-dashes combined with shallowness of meaning combined with unnecessary overdramatisations.

Re: Cloudflare outage on November 18, 2025 post mortem

#598
post #518
post #498

Earlier quoted context omitted.

Costs to architect systems that serve millions of request daily have gone down . Not up. Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client. People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)

> Costs to architect systems that serve millions of request daily have gone down. Not up. I never said serving millions of requests is more expensive. Protecting your servers is more expensive. > Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client. HN uses Cloudflare. You're making my point for me. If you included the fixed costs that Cloud…

> HN uses Cloudflare

From the ping output, I can see HN is using m5hosting.com. This is why HN was up yesterday, even though everything on CF was down.

> Writing high-throughput web applications is easier than ever. Hosting them on the open web is harder than ever.

Writing proper high-throughput applications was never easy and will never be. It is a little bit easier because we have highly optimized tools like nginx or nodejs so we can offset critical parts. And hosting is "harder than ever" if you complicate the matter, which is a quite common pattern these days. I saw people running monstrosities to serve some html & js in the name of redundancy. You'd be surprised how much a single bare-metal (hell, even a proper VM from DigitalOcean or Vultr) can handle.

Re: Cloudflare outage on November 18, 2025 post mortem

#599

Earlier quoted context omitted.

I think defence against a DDOS against your network is the best reason for a quick rollout

This was not about DDoS defense but the Bot Management feature, which is a paid Enterprise-only feature not enabled by default to block automated requests regardless of whether an attack is going on. https://developers.cloudflare.com/bots/get-started/bot-manag...

So if you didn't enable it your stuff would work?

Re: Cloudflare outage on November 18, 2025 post mortem

#600
post #525

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code. And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.

  [lints.clippy]
  dbg_macro = "deny"
  unwrap_used = "deny"
  expect_used = "deny"
Post reply on HN