Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

761–770 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#761

Post by senbrow »

anonymous908213 wrote: post #597
deadcore wrote: post #526

Earlier quoted context omitted.

Out of interest... apart from the em dash, how else can you tell it's an LLM response? What are the telltales signs?

This particular excerpt is reeking of it with pretty much every line. I'll point out the patterns in the English translation, but all of these patterns apply cross-language. > classic chain reaction triggered by “hidden assumptions + configuration chains” "Classic/typical "x + y"", particularly when diagnosing an issue. This one is a really easy tell because humans, on aggregate, do not use quotation marks like this.…

Thanks for writing such a detailed reply about this.

Like GP I'm not very good at spotting these patterns yet, so explicit real-world examples go a long way.

Re: Cloudflare outage on November 18, 2025 post mortem

#763

Post by rkomorn »

zf00002 wrote: post #750

As an IT person, I wonder what it's like to work for a company like this. Where presumably IT stuff has a priority. Unlike the companies I've worked for where IT takes a backseat to everything until something goes wrong. Company I work had a huge new office built, with the plan it would be big enough for future growth, yet despite repeated attempts to reserve a larger space, our server room and infrastructure is actu…

As a former CF employee, I'd say it's a mixed bag.

There are plenty of resources , yet it's somehow never enough. You do tons of pretty amazing things with pretty amazing tools that also have notable shortcomings.

You're surround by smart people who do lots of great work, but you also end up in incident reviews where you find facepalm-y stuff. Sometimes you even find out it was a known corner case that was deemed too unlikely to prioritize.

The last incident for my team that I remember dealing with there ended up with my coworker and I realizing the staging environment we'd taken down hours earlier was actually the source of data for a production dashboard, so we'd lost some visibility and monitoring for a bit.

I've also worked at Facebook (pre-Meta days) and at Datadog, and I'd say it was about the same. Most things are done quite well, but so much stuff is happening that you still end up with occasional incidents that feel like they shouldn't have happened.

Re: Cloudflare outage on November 18, 2025 post mortem

#764

Post by NewJazz »

thesh4d0w wrote: post #71
yen223 wrote: post #57

Earlier quoted context omitted.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

The person who posted both this blog article and the hacker news post, is Matthew Prince, one of highly technical billionaire founders of cloudflare. I'm sure if he wants something to happen, it happens.

I'm sure he wanted traffic to flow for those few hours but it didn't :p

Re: Cloudflare outage on November 18, 2025 post mortem

#765

Post by frumplestlatz »

burntsushi wrote: post #729
frumplestlatz wrote: post #549

Earlier quoted context omitted.

I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality. Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled. If you need the value, you need to handle the case that it’s not available explicitly. Y…

The blog linked in the GP anticipates this rebuttal and already addresses it. Your argument also implies that things like `slice[i]` are never okay.

`slice[i]` is also a hole in the type system, but at least it’s generally relying on a local invariant, immediate to the surrounding context, that does not require lying about invariants across your API surface.

The blog post doesn’t address the issue, it simply pretends it’s not a real problem.

Also from the post: “If we were to steelman advocates in favor of this style of coding, then I think the argument is probably best limited to certain high reliability domains. I personally don’t have a ton of experience in said domains …”

Enough said.

Re: Cloudflare outage on November 18, 2025 post mortem

#768

Post by asenchi »

tptacek wrote: post #50
eastdakota wrote: post #35

Earlier quoted context omitted.

Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.

Richard Cook #18 (and #10) strikes again! https://how.complexsystems.fail/#18 It'd be fun to read more about how you all procedurally respond to this (but maybe this is just a fixation of mine lately). Like are you tabletopping this scenario, are teams building out runbooks for how to quickly resolve this, what's the balancing test for "this needs a functional change to how our distributed systems work" vs. "instead…

This document by Dr. Cook remains _the standard_ for systems failure. Thank you for bringing it into the discussion.

Re: Cloudflare outage on November 18, 2025 post mortem

#769

Post by grogers »

ergocoder wrote: post #478
fredoralive wrote: post #467

Earlier quoted context omitted.

A failed config load probably shouldn't be a fatal error if a valid config is already loaded?

Hard to say. Why would you load a new config if a valid config is already loaded? Maybe the new config has a new update. Who knows? Do we want to keep operating on the old config? Maybe maybe not. But operating on old config when you don't want to is definitely worse.

Of course it depends on the situation. But I don't see how you could think that in this case, crashing is better than stale config.

Crashing on a config update is usually only done if it could cause data corruption if the configs aren't in sync. That's obviously not the case here since the updates (although distributed in real time) are not coupled between hosts. Such systems usually are replicated state machines where config is totally ordered relative to other commands. Example: database schema and write operations (even here the way many databases are operated they don't strongly couple the two).

Re: Cloudflare outage on November 18, 2025 post mortem

#770

Post by bombela »

mike_hearn wrote: post #760
bombela wrote: post #490

Earlier quoted context omitted.

This thread warms my heart. Rust has set a new baseline that many and myself now take for granted. We are now discussing what can be done to improve code correctness beyond memory and thread safety. I am excited for what is to come.

Really not! This is a huge faceplant for writing things in Rust. If they had been writing their code in Java/Kotlin instead of Rust, this outage either wouldn't have happened at all (a failure to load a new config would have been caught by a defensive exception handler), or would have been resolved in minutes instead of hours. The most useful thing exceptions give you is not static compile time checking, it's the sta…

https://doc.rust-lang.org/std/backtrace/index.html#environme...

tldr: Capturing a backtrace can be a quite expensive runtime operation, so the environment variables allow either forcibly disabling this runtime performance hit or allow selectively enabling it in some programs.

By default it is disabled in release mode.

Post reply on HN