Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

341–350 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#341

Post by otterley »

hedora wrote: post #291
otterley wrote: post #269

Earlier quoted context omitted.

It wasn't worth thinking about. I'm not going to defend myself against arguments and absolute claims I didn't make. The key word here is mitigation, not perfection.

> If your AWS service is properly regionalized, that’s the minimum amount of cellular architecture required Amazon has had multi-region outages due to pushing bad configs, so it’s extremely difficult to believe whatever you are proposing solves that exact problem by relying on multi-regions. Come to think of it, Cloudflare’s outage today is another good counterexample.

It has been a very, very long time since AWS had a simultaneous failure across multiple regions. Even customers impacted by the loss of Route 53 control plane functionality in last month’s us-east-1 were able to gracefully fail over to a backup region if they configured failover records in advance, had Application Recovery Controller set up, or fronted their APIs or websites with Global Accelerator.

Customers survive incidents on a daily basis by failing over across regions (even in the absence of an AWS regional failure, they can fail due to a bad deployment or other cause). The reason you don’t hear about it is because it works.

Re: Cloudflare outage on November 18, 2025 post mortem

#342

Post by aw1621107 »

ashishb wrote: post #233
tristan-morris wrote: post #60

Earlier quoted context omitted.

And rust, but they chose to panic on the error condition. Wild.

> And rust, but they chose to panic on the error condition. Wild. unwrap() implicitly panic-ed, right?

I don't think "implicitly panicked" is an accurate description since unwrap()'s entire reason for existing is to panic if you unwrap an error condition. If you use unwrap(), you're explicitly opting into the panicking behavior.

I suppose another way to think about it is that Result is somewhat analogous to Java's checked exceptions - you can't get the T out unless you say what to do in the case of the E/checked exception. unwrap() in this context is equivalent to wrapping the checked exception in a RuntimeException and throwing that.

Re: Cloudflare outage on November 18, 2025 post mortem

#343

Post by bigstrat2003 »

Dylan16807 wrote: post #152
Rohansi wrote: post #144

Earlier quoted context omitted.

> when you do unwrap() you're saying the Result should NEVER fail Returning a Result by definition means the method can fail.

> Returning a Result by definition means the method can fail. No more than returning an int by definition means the method can return -2.

> No more than returning an int by definition means the method can return -2.

What? Returning an int does in fact mean that the method can return -2. I have no idea what your argument is with this, because you seem to be disagreeing with the person while actually agreeing with them.

Re: Cloudflare outage on November 18, 2025 post mortem

#344

Post by Feathercrown »

vlovich123 wrote: post #247
hedora wrote: post #186

Earlier quoted context omitted.

Catching panic probably isn’t a great idea if there’s any unsafe code in the system. (Do the unsafe blocks really maintain heap invariants if across panics?)

Unsafe blocks have nothing to do with it. Yes - they maintain all the same invariants as safe blocks or those unsafe blocks are unsound regardless of panics. But there’s millions of way to architect this (eg a supervisor process that notices which layer in FL2 is crashing and just completely disables that layer when it starts up the proxy again. There’s challenges here because then you have to figure out what constit…

Incremental config changes sounds like it could lead to a LOT of bugs

Re: Cloudflare outage on November 18, 2025 post mortem

#345

Post by bobbylarrybobby »

AgentME wrote: post #214
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a n…

I don't know too much about how the feature file distribution works but in the event of failure to read a new file, wouldn't logging the failure and sticking with the previous version of the file be preferable?

Re: Cloudflare outage on November 18, 2025 post mortem

#346

Post by echelon »

wrs wrote: post #238
tux3 wrote: post #165

Earlier quoted context omitted.

Usually you'd want to write almost all your slice or other container iterations with iterators, in a functional style. For the 5% of cases that are too complex for standard iterators? I never bother justifying why my indexes are correct, but I don't see why not. You very rarely need SAFETY comments in Rust because almost all the code you write is safe in the first place. The language also gives you the tool to avoid…

I didn't restate the context from the code we're discussing: it must not panic. If you don't care if the code panics, then go ahead and unwrap/expect/index, because that conforms to your chosen error handling scheme. This is fine for lots of things like CLI tools or isolated subprocesses, and makes review a lot easier. So: first, identify code that cannot be allowed to panic. Within that code, yes, in the rare case t…

Cargo needs to grow a label for crates that provably do not panic. (Neverminding allocations and things outside our control flow.)

I want to ban crates that panic from my dependency chain.

The language could really use an extra set of static guarantees around this. I would opt in.

Re: Cloudflare outage on November 18, 2025 post mortem

#347

Post by bigstrat2003 »

echelon wrote: post #327
kibwen wrote: post #323

Earlier quoted context omitted.

The `unsafe` keyword means something specific in Rust, and panicking isn't unsafe by Rust's definition. Sometimes avoiding partial functions just isn't feasible, and an unwrap (or whatever you want to call the method) is a way of providing a (runtime-checked) proof to the compiler that the function is actually total.

Panics should be explicit, not implicit. unwrap() should effectively work as a Result where the user must manually invoke a panic in the failure branch. Make special syntax if a match and panic is too much boilerplate. This is like an implicit null pointer exception that cannot be statically guarded against. I want a way to statically block any crates doing this from my dependency chain.

unwrap is explicit.

Re: Cloudflare outage on November 18, 2025 post mortem

#348

Post by tick_tock_tick »

lmm wrote: post #128
dzonga wrote: post #54

> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually…

> Rust won't save you from the usual programming mistake. Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like. No language will prevent you from writing broken code if you're determined to do so, and no language is impossible to write correct code in if you…

> Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like.

No one treats it like that and nearly every Rust project is filled with unwraps all over the place even in production system like Cloudflare's.

Re: Cloudflare outage on November 18, 2025 post mortem

#349

Post by Terr_ »

bigstrat2003 wrote: post #315
Terr_ wrote: post #276

Earlier quoted context omitted.

"Checked Exceptions Are Actually Good" gang, rise up! :p I think adoption would have played out very different if there had only been some more syntactic-sugar. For example, an easy syntax for saying: "In this method, any (checked) DeepException e that bubbles up should immediately be replaced by a new (checked) MylayerException(e) that contains the original one as a cause. We might still get lazy programmers making…

I'm with you! Checked exceptions are actually good and the hate for them is super short sighted. The exact same criticisms levied at checked exceptions apply to static typing in general, but people acknowledge the great value static types have for preventing errors at compile time. Checked exceptions have that same value, but are dunked on for some reason.

Yeah, in both cases it's a layering situation, where it's the duty of your code to decide what layers of abstraction need to be be bridged, and to execute on that decision. Translating/wrapping exception-types from deeper functions is the same as translating/wrapping return-types the same places.

I think it comes down to a psychological or use-case issue: People hate thinking about errors and handling them, because it's that hard stuff that always consumes more time than we'd like to think. Not just digitally, but in physical machines too. It's also easier to put off "for later."

Re: Cloudflare outage on November 18, 2025 post mortem

#350

Post by wrs »

dist1ll wrote: post #287
hansvm wrote: post #277

Earlier quoted context omitted.

> graph-heavy code Could you share some more details, maybe one fully concrete scenario? There are lots of techniques, but there's no one-size-fits-all solution.

Sure, these days I'm mostly working on a few compilers. Let's say I want to make a fixed-size SSA IR. Each instruction has an opcode and two operands (which are essentially pointers to other instructions). The IR is populated in one phase, and then lowered in the next. During lowering I run a few peephole and code motion optimizations on the IR, and then do regalloc + asm codegen. During that pass the IR is mutated a…

And it's fine for a compiler to panic when it violates an assumption. Not so with the Cloudflare code under discussion.
Post reply on HN