Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

411–420 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#411

Post by sanjitb »

perching_aix wrote: post #381
sanjitb wrote: post #372

cloudflare: > Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. also cloudflare: > The Cloudflare Dashboard was also impacted due to both Workers KV being used internally and Cloudflare Turnstile being deployed as part…

I believe you're mistakenly equating Cloudflare's status page with the Cloudflare Dashboard? They're not the same thing. Cloudflare's status page: https://www.cloudflarestatus.com/ Cloudflare Dashboard: https://dash.cloudflare.com/

thanks for clarifying! i guess then they never explained why the status page went down, even though it's supposed to be running on independent infrastructure.

Re: Cloudflare outage on November 18, 2025 post mortem

#412

Post by lenkite »

bigstrat2003 wrote: post #315
Terr_ wrote: post #276

Earlier quoted context omitted.

"Checked Exceptions Are Actually Good" gang, rise up! :p I think adoption would have played out very different if there had only been some more syntactic-sugar. For example, an easy syntax for saying: "In this method, any (checked) DeepException e that bubbles up should immediately be replaced by a new (checked) MylayerException(e) that contains the original one as a cause. We might still get lazy programmers making…

I'm with you! Checked exceptions are actually good and the hate for them is super short sighted. The exact same criticisms levied at checked exceptions apply to static typing in general, but people acknowledge the great value static types have for preventing errors at compile time. Checked exceptions have that same value, but are dunked on for some reason.

Checked exceptions in theory were good, but Java simply did not add facilities to handle or support them well in many APIs. Even the new API's in Java - Streams, etc do not support checked exceptions.

Re: Cloudflare outage on November 18, 2025 post mortem

#413

Post by thatoneengineer »

The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen".

The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they weren't.

The slowness to root cause: sheer bad luck, with the status page down and Azure's DDoS yesterday all over the news.

The broken SQL: this is the one that I'd be up in arms about if I worked for Cloudflare. For a system with the power to roll out config to ~all of prod at once while bypassing a lot of the usual change tracking, having this escape testing and review is a major miss.

Re: Cloudflare outage on November 18, 2025 post mortem

#414

Post by lenkite »

pdimitar wrote: post #179
cvhc wrote: post #122

Earlier quoted context omitted.

Some languages and style guides simply forbid throwing exceptions without catching / proper recovery. Google C++ bans exceptions and the main mechanism for propogating errors is `absl::Status` which the caller has to check. Not familiar with Rust but it seems unwrap is such a thing that would be banned.

There are even lints for this but people get impatient and just override them or fight for them to no longer be the default. As usual: people problem, not a tech problem. In the last years a lot of strides have been made. But people will be people.

Linting is not good enough. The compiler should refuse to compile the code without it marked with an explicit annotation. Too much Rust code is panic happy since using casual use of `unwrap` is perma-etched into everyone's minds by the amount of demo code out there using unwrap.

Re: Cloudflare outage on November 18, 2025 post mortem

#415

Post by anonymous908213 »

slanterns wrote: post #379
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> Today, many friends pinged me saying Cloudflare was down. As a core developer of the first generation of Cloudflare FL, I'd like to share some thoughts. > This wasn't an attack, but a classic chain reaction triggered by “hidden assumptions + configuration chains” — permission changes exposed underlying tables, doubling the number of lines in the generated feature file. This exceeded FL2's memory preset, ultimately…

This tweet thread invokes genuine despair in me. Do we really have to outsource even our tweets to LLMs? Really? I mean, I get spambots and the like tweeting mass-produced slop. But what compels a former engineer of the company in question to offer LLM-generated "insight" to the outage? Why? For what purpose?

* For clarity, I am aware that the original tweets are written in Chinese, and they still have the stench of LLM writing all over them; it's not just the translation provided in the above comment.

Re: Cloudflare outage on November 18, 2025 post mortem

#416

Post by wrs »

lenkite wrote: post #408
wrs wrote: post #238

Earlier quoted context omitted.

I didn't restate the context from the code we're discussing: it must not panic. If you don't care if the code panics, then go ahead and unwrap/expect/index, because that conforms to your chosen error handling scheme. This is fine for lots of things like CLI tools or isolated subprocesses, and makes review a lot easier. So: first, identify code that cannot be allowed to panic. Within that code, yes, in the rare case t…

What about memory allocation - how will you stop that from panicking ? `Vec::resize` will always panic in Rust. And this is just one example out of thousands in the Rust stdlib. Unless the language addresses no-panic in its governing design or allows try-catch, not sure how you go about this.

That is slowly being addressed, but meanwhile it’s likely you have a reliable upper bound on how much heap your service needs, so it’s a much smaller worry. There are also techniques like up-front or static allocation if you want to make more certain.

Re: Cloudflare outage on November 18, 2025 post mortem

#417

Post by wonnage »

anonymous908213 wrote: post #415
slanterns wrote: post #379

Earlier quoted context omitted.

> Today, many friends pinged me saying Cloudflare was down. As a core developer of the first generation of Cloudflare FL, I'd like to share some thoughts. > This wasn't an attack, but a classic chain reaction triggered by “hidden assumptions + configuration chains” — permission changes exposed underlying tables, doubling the number of lines in the generated feature file. This exceeded FL2's memory preset, ultimately…

This tweet thread invokes genuine despair in me. Do we really have to outsource even our tweets to LLMs? Really? I mean, I get spambots and the like tweeting mass-produced slop. But what compels a former engineer of the company in question to offer LLM-generated "insight" to the outage? Why? For what purpose? * For clarity, I am aware that the original tweets are written in Chinese, and they still have the stench of…

It’s kinda funny that the “not …, but…” + em dash slop signifiers translate directly to Chinese “不是。。。而是。。。” and double-width em dash

Re: Cloudflare outage on November 18, 2025 post mortem

#418

Post by wrs »

littlestymaar wrote: post #402
wrs wrote: post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

> In production code an unwrap or expect should be reviewed exactly like a panic. An unwrap should never make it to production IMHO. It's fine while prototyping, but once the project gets closer to production it's necessary to just grep `uncheck` in your code and replace those that can happen with a proper error management and replace those that cannot happen with `expect`, with a clear justification of why they cann…

I would say, sure, if you feel the same way about panic calls making to production. In other words, review all of them the same way. Because writing unwrap/expect is exactly the same as writing “if error, panic”.

Re: Cloudflare outage on November 18, 2025 post mortem

#420

Post by assbuttbuttass »

lenkite wrote: post #408
wrs wrote: post #238

Earlier quoted context omitted.

I didn't restate the context from the code we're discussing: it must not panic. If you don't care if the code panics, then go ahead and unwrap/expect/index, because that conforms to your chosen error handling scheme. This is fine for lots of things like CLI tools or isolated subprocesses, and makes review a lot easier. So: first, identify code that cannot be allowed to panic. Within that code, yes, in the rare case t…

What about memory allocation - how will you stop that from panicking ? `Vec::resize` will always panic in Rust. And this is just one example out of thousands in the Rust stdlib. Unless the language addresses no-panic in its governing design or allows try-catch, not sure how you go about this.

In TFA they mentioned they preallocate all the memory up front
Post reply on HN