Cloudflare outage on November 18, 2025 post mortem • Hacker News Board

Re: Cloudflare outage on November 18, 2025 post mortem

#201

Post by tonyhart7 » Wed, Nov 19, 2025, 1:47 AM UTC

pdimitar wrote: post #179

cvhc wrote: post #122

Earlier quoted context omitted.

Some languages and style guides simply forbid throwing exceptions without catching / proper recovery. Google C++ bans exceptions and the main mechanism for propogating errors is `absl::Status` which the caller has to check. Not familiar with Rust but it seems unwrap is such a thing that would be banned.

There are even lints for this but people get impatient and just override them or fight for them to no longer be the default. As usual: people problem, not a tech problem. In the last years a lot of strides have been made. But people will be people.

and people make mistake

at some point machine would be better in coding because well machine code is machine instruction task

same like chess, engine is better than human grandmaster because its solvable math field

coding is no different

Re: Cloudflare outage on November 18, 2025 post mortem

#202

Post by ademarre » Wed, Nov 19, 2025, 1:48 AM UTC

I integrated Turnstile with a fail-open strategy that proved itself today. Basically, if the Turnstile JS fails to load in the browser (or in a few specific frontend error conditions), we allow the user to submit the web form with a dummy challenge token. On the backend, we process the dummy token like normal, and if there is an error or timeout checking Turnstile's siteverify endpoint, we fail open.

Of course, some users were still blocked, because the Turnstile JS failed to load in their browser but the subsequent siteverify check succeeded on the backend. But overall the fail-open implementation lessened impact to our customers nonetheless.

Fail-open with Turnstile works for us because we have other bot mitigations that are sufficient to fall back on in the event of a Cloudflare outage.

Re: Cloudflare outage on November 18, 2025 post mortem

#203

Post by ants_everywhere » Wed, Nov 19, 2025, 1:49 AM UTC

otterley wrote: post #19

> work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditio…

it's always a config push. people rollout code slowly but don't have the same mechanisms for configs. But configs are code, and this is a blind spot that causes an outsized percentage of these big outages.

Re: Cloudflare outage on November 18, 2025 post mortem

#204

Post by gpm » Wed, Nov 19, 2025, 1:49 AM UTC

cherryteastain wrote: post #193

gpm wrote: post #181

Earlier quoted context omitted.

Unwrap is used in places where in C++ you would just have undefined behavior. It wouldn't make any more sense to blanket ban it than it would ban ever dereferencing a pointer just in case its null - even if you just checked that it wasn't null.

Rust's Result is the same thing as C++'s std::expected. How is calling std::expected::value undefined behaviour?

Rust's foo: Option is rust's rough equivalent to C++'s const T* foo. The C++ *foo is equivalent to the rust unsafe{ *foo.unwrap_unchecked() }, or in safe code *foo.unwrap() (which changes the undefined behavior to a panic).

Rust's unwrap isn't the same as std::expected::value. The former panics - i.e. either aborts the program or unwinds depending on context and is generally not meant to be handled. The latter just throws an exception that is generally expected to be handled. Panics and exceptions use similar machinery (at least they can depending on compiler options) but they are not equivalent - for example nested panics in destructors always abort the program.

In code that isn't meant to crash `unwind` should be treated as a sign saying that "I'm promising that this will never happen", but just like in C++ where you promise that pointers you deference are valid and signed integers you add don't overflow making promises like that is a necessary part of productive programming.

Re: Cloudflare outage on November 18, 2025 post mortem

#205

Post by adzm » Wed, Nov 19, 2025, 1:49 AM UTC

nine_k wrote: post #143

arccy wrote: post #89

Earlier quoted context omitted.

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

At Facebook they name certain "escape hatch" functions in a way that inescapably make them look like a GIANT EYESORE. Stuff like DANGEROUSLY_CAST_THIS_TO_THAT, or INVOKE_SUPER_EXPENSIVE_ACTION_SEE_YOU_ON_CODE_REVIEW. This really drives home the point that such things must not be used except in rare extraordinary cases. If unwrap() were named UNWRAP_OR_PANIC(), it would be used much less glibly. Even more, I wish ther…

> make them look like a GIANT EYESORE

React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED comes to mind. I did have to reach to this before, but it certainly works for keeping this out of example code and other things like reading other implementations without the danger being very apparent.

At some point it was renamed to __CLIENT_INTERNALS_DO_NOT_USE_OR_WARN_USERS_THEY_CANNOT_UPGRADE which is much less fun.

Re: Cloudflare outage on November 18, 2025 post mortem

#206

Post by yoyohello13 » Wed, Nov 19, 2025, 1:50 AM UTC

nomel wrote: post #178

yoyohello13 wrote: post #163

Earlier quoted context omitted.

Probably not, since errors as values are way better than exceptions.

How so? An exception is a value that's given the closest, conceptually appropriate, point that was decided to handle the value, allowing you to keep your "happy path" as clean code, and your "exceptional circumstances" path at the level of abstraction that makes sense. It's way less book-keeping with exceptions, since you, intentionally, don't have to write code for that exceptional behavior, except where it makes se…

There are many many pages of text discussing this topic, but having programmed in both styles, exceptions make it too easy for programmer to simply ignore them. Errors as values force you to explicitly handle it there, or toss it up the stack. Maybe some other languages have better exception handling but in Python it’s god awful. In big projects you can basically never know when or how something can fail.

Re: Cloudflare outage on November 18, 2025 post mortem

#207

Post by smj-edison » Wed, Nov 19, 2025, 1:50 AM UTC

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Isn't the point of this article that pieces of infrastructure don't go down to root causes, but due to bad combinations of components that are correct individually? After reading "engineering a safer world", I find root cause analysis rather reductionistic, because it wasn't just an unwrap, it was that the payload was larger than normal, because of a query that didn't select by database, because a clickhouse made more databases visible. Hard to say "it was just due to an unwrap" imo. Especially in terms of how to fix an issue going forwards. I think the article lists a lot of good ideas, that aren't just "don't unwrap", like enabling more global kill switches for features, or eliminating the ability for core dumps or other error reports to overwhelm system resources.

Re: Cloudflare outage on November 18, 2025 post mortem

#208

Post by pdimitar » Wed, Nov 19, 2025, 1:50 AM UTC

While I heavily frown upon using `unwrap` and `expect` in Rust code and make sure to have Clippy tell me about every single usage of them, I also understand that without them Rust might have been seen as an academic curiosity language.

They are escape hatches. Without those your language would never take off.

But here's the thing. Escape hatches are like emergency exits. They are not to be used by your team to go to lunch in a nearby restaurant.

---

Cloudflare should likely invest in better linting and CI/CD alerts. Not to mention isolated testing i.e. deploy this change only to a small subset and monitor, and only then do a wider deployment.

Hindsight is 20/20 and we can all be smartasses after the fact of course. But I am really surprised because lately I am only using Rust for hobby projects and even I know I should not use `unwrap` and `expect` beyond the first iteration phases.

---

I have advocated for this before but IMO Rust at this point will benefit greatly from disallowing those unsafe APIs by default in release mode. Though I understand why they don't want to do it -- likely millions of CI/CD pipelines will break overnight. But in the interim, maybe a rustc flag we can put in our `Cargo.toml` that enables such a stricter mode? Or have that flag just remove all the panicky API _at compile time_ though I believe this might be a Gargantuan effort and is likely never happening (sadly).

In any case, I would expect many other failures from Cloudflare but not _this_ one in particular.

Re: Cloudflare outage on November 18, 2025 post mortem

#209

Post by cj » Wed, Nov 19, 2025, 1:51 AM UTC

ademarre wrote: post #202

I integrated Turnstile with a fail-open strategy that proved itself today. Basically, if the Turnstile JS fails to load in the browser (or in a few specific frontend error conditions), we allow the user to submit the web form with a dummy challenge token. On the backend, we process the dummy token like normal, and if there is an error or timeout checking Turnstile's siteverify endpoint, we fail open. Of course, some…

So to bypass captcha all a user has to do is block the script from loading? I can see that working but only for attacks that aren’t targeted?

Re: Cloudflare outage on November 18, 2025 post mortem

#210

Post by jofzar » Wed, Nov 19, 2025, 1:51 AM UTC

eastdakota wrote: post #95

yen223 wrote: post #57

Earlier quoted context omitted.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…

> I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did

Damn corporate karma farming is ruthless, only a couple minute SLA before taking ownership of the karma. I guess I'm not built for this big business SLA.