Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

881–890 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#881

Post by EvanAnderson »

tptacek wrote: post #878
EvanAnderson wrote: post #875

Earlier quoted context omitted.

I'm sorry to belabor this but I'm genuinely not understanding what you're saying in this reply. I haven't operated large scale systems. I'm just an IT generalist and casual coder. I acknowledge I'm too inexperienced to even know what I don't know re: running large systems. I read the parent poster as broadly suggesting configuration updates should have fitness tests applied and be deployed to minimize the blast radiu…

The things you do to safeguard the rollout of a configuration file change are not the same as the things you do to reliably propagate changes that might happen many times per second. What's irritating to me are the claims that there's nothing distinguishing real time control plane state changes and config files. Most of us have an intuition for how they'd do a careful rollout of a config file change. That intuition d…

Thanks for the reply.

I took the "this sounds like Crowdstrike" tack for two reasons. The write-up characterized this update as an every five minutes process. The update, being a file of rules, felt analogous in format to the Crowdstrike signature database.

I appreciate the OSPF analogy. I recognize there are portions of these large systems that operate more like a routing protocol (with updates being unpredictable in velocity or time of occurrence). The write-up didn't make this seem like one of those. This seemed a lot more like a traditional daemon process receiving regular configuration updates and crashing on a bad configuration file.

Re: Cloudflare outage on November 18, 2025 post mortem

#882

Post by tptacek »

EvanAnderson wrote: post #881
tptacek wrote: post #878

Earlier quoted context omitted.

The things you do to safeguard the rollout of a configuration file change are not the same as the things you do to reliably propagate changes that might happen many times per second. What's irritating to me are the claims that there's nothing distinguishing real time control plane state changes and config files. Most of us have an intuition for how they'd do a careful rollout of a config file change. That intuition d…

Thanks for the reply. I took the "this sounds like Crowdstrike" tack for two reasons. The write-up characterized this update as an every five minutes process. The update, being a file of rules, felt analogous in format to the Crowdstrike signature database. I appreciate the OSPF analogy. I recognize there are portions of these large systems that operate more like a routing protocol (with updates being unpredictable i…

It is possible that any number of things people on this thread have called out are, in fact, the right move for the system Cloudflare built (it's hard to know without knowing more about the system, and my intuition for their system is also faulty because I irrationally hate periodic batch systems like these).

Most of what I'm saying is:

(1) Looking at individual point failures and saying "if you'd just fixed that you wouldn't have had an incident" is counterproductive; like Mr. Oogie-Boogie, every big distributed system is made of bugs. In fact, that's true of literally every complex system, which is part of the subtext behind Cook[1].

(2) I think people are much too quick to key in on the word "config" and just assume that it's morally indifferentiable from source code, which is rarely true in large systems like this (might it have been here? I don't know.) So my eyes twitch like Louise Belcher's when people say "config? you should have had a staged rollout process!" Depends on what you're calling "config"!

[1] https://howcomplexsystems.fail/

Re: Cloudflare outage on November 18, 2025 post mortem

#883

Post by inferiorhuman »

jandrewrogers wrote: post #737
inferiorhuman wrote: post #521

Earlier quoted context omitted.

Dunno, I think the alternatives have their own pretty significant downsides. All would require front loading more in-depth understanding of error handling and some would just be quite a bit more verbose. IMO making unwrap a clippy lint (or perhaps a warning) would be a decent start. Or maybe renaming unwrap.

This strikes me as a culture issue more than one of language. A tenet of systems code is that every possible error must be handled explicitly and exhaustively close to the point of occurrence. It doesn’t matter if it is Rust, C, etc. Knowing how to write systems code is unrelated to knowing a systems language. Rust is a systems language but most people coming into Rust have no systems code experience and are “holding…

  A tenet of systems code is that every possible error must be handled
  explicitly and exhaustively close to the point of occurrence.
All the more reason it doesn't really belong in examples for third party libraries.

Re: Cloudflare outage on November 18, 2025 post mortem

#884

Post by vbezhenar »

loglog wrote: post #867
vbezhenar wrote: post #585

Earlier quoted context omitted.

I've found both methods very useful. I'm using `get()` when I've checked that the value is present and I don't expect any exceptions. I'm using `orElseThrow()` when I actually expect that value can be absent and throwing is fine. Something like if (userOpt.isPresent()) { var user = userOpt.get(); var accountOpt = accountRepository.selectAccountOpt(user.getId()); var account = accountOpt.orElseThrow(); } Idea checks i…

While the `Optional` API is generally pretty inconvenient (compared e.g. to Kotlin), it does offer the more precise `ifPresent`.

Java lambdas are terrible so I usually avoid them. There's no reason to invent new methods, when language already has corresponding statements.

Re: Cloudflare outage on November 18, 2025 post mortem

#885

Post by tick_tock_tick »

antihero wrote: post #710
CamouflagedKiwi wrote: post #593

Earlier quoted context omitted.

It's literally not, Rust tutorials are littered with `.unwrap()` calls. It might be Rust 102, but the first impression given is that the language is surprisingly happy with it.

https://doc.rust-lang.org/book/ch09-02-recoverable-errors-wi... If you haven't read the Rust Book at least, which is effectively Rust 101, you should not be writing Rust professionally. It has a chapter explaining all of this.

> In production-quality code, most Rustaceans choose expect rather than unwrap and give more context about why the operation is expected to always succeed. That way, if your assumptions are ever proven wrong, you have more information to use in debugging.

I didn't read anything in that section about unwrap/expect that it shouldn't be used in production code. If anything I read it as perfectly acceptable.

Re: Cloudflare outage on November 18, 2025 post mortem

#887

Post by 9rx »

tuetuopay wrote: post #627
9rx wrote: post #548

Earlier quoted context omitted.

The takeaway here isn’t about Rust itself, but that the Rust marketing crew’s claims that we constantly read on HN and elsewhere about the Result type magically saving you from making mistakes is not a good message to send.

They would also tell you that .unwrap() has no place in production code, and should receive as much scrutiny as an `unsafe` block in code review :) The point of option is the crash path is more verbose and explicit than the crash-free path. It takes more code to check for NULL in C or nil in Go; it takes more code in Rust to not check for Err.

1. They don’t. There is presumably some hypothetical world where they would tell you if you start asking questions, but nobody buying into the sales pitch ever asks questions.

2. You’re getting confused by technology again. This isn’t about technology.

Re: Cloudflare outage on November 18, 2025 post mortem

#888

Post by ergocoder »

grogers wrote: post #769
ergocoder wrote: post #478

Earlier quoted context omitted.

Hard to say. Why would you load a new config if a valid config is already loaded? Maybe the new config has a new update. Who knows? Do we want to keep operating on the old config? Maybe maybe not. But operating on old config when you don't want to is definitely worse.

Of course it depends on the situation. But I don't see how you could think that in this case, crashing is better than stale config. Crashing on a config update is usually only done if it could cause data corruption if the configs aren't in sync. That's obviously not the case here since the updates (although distributed in real time) are not coupled between hosts. Such systems usually are replicated state machines whe…

Because stale config could easily go unnoticed for a long time.

Crashing is generally better than behaving incorrectly due to stale configs. Because the problem would get fixed faster.

Re: Cloudflare outage on November 18, 2025 post mortem

#889

Post by vlovich123 »

antihero wrote: post #683
hedora wrote: post #186

Earlier quoted context omitted.

Catching panic probably isn’t a great idea if there’s any unsafe code in the system. (Do the unsafe blocks really maintain heap invariants if across panics?)

The unwrap should be replaced by code that creates enough alerting to make a P0 incidident from their canary deployment immediately. OR even, the bot code crashing should itself be generating alerts. Canary deployment would be automatically rolled back until P0 incident resolved. All of this could probably have happened and contained at their scale in less than a minute as they would likely generate enough "omg the p…

Agreed - a big question why the file wasn’t test driven in staging and progressively rolled out. And also what alerting was missing within FL2 that they couldn’t pinpoint the unwrap instantly.

Re: Cloudflare outage on November 18, 2025 post mortem

#890

Post by bombela »

mike_hearn wrote: post #781
bombela wrote: post #770

Earlier quoted context omitted.

https://doc.rust-lang.org/std/backtrace/index.html#environme... tldr: Capturing a backtrace can be a quite expensive runtime operation, so the environment variables allow either forcibly disabling this runtime performance hit or allow selectively enabling it in some programs. By default it is disabled in release mode.

It's one of the problems with using result types. You don't distinguish between genuinely exceptional events and things that are expected to happen often on hot paths, so the runtime doesn't know how much data to collect.

panic is the exceptional event. It so happens that rust doesn't print a stacktrace in release unless configured to do so.

Similarly, capturing a stack trace in a error type (within a Result for example) is perfectly possible. But this is a choice left to the programmer, because capturing a trace is not cheap.

Post reply on HN