Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

941–950 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#941

Post by auto »

eastdakota wrote: post #95
yen223 wrote: post #57

Earlier quoted context omitted.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…

I'm not sure I've ever read something from someone so high up in a company that gave me such a strong feeling for "I'd like to work for these people". If job posts could be so informal and open ended, this post could serve as one in the form of a personality fit litmus test.

Re: Cloudflare outage on November 18, 2025 post mortem

#942

Post by AdieuToLogic »

jcalvinowens wrote: post #317
AdieuToLogic wrote: post #309

Earlier quoted context omitted.

>> This is the multi-million dollar .unwrap() story. > That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same. Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assu…

> Instead, the routine likely should have worked within `Result`. But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same. Maybe you're saying "it's better to be explicit", as a broad generalization I don't disagree with that. But that has nothing to do with the actual bug here, which was that the invariant failed. How they choose to implement checking and failing the invar…

>> Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assuming `features.append_with_names` would never fail. Instead, the routine likely should have worked within `Result`.

> But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same.

I agree it is an error, but disagree that it should be a fatal error at that location. The reason being is the method defining the offending `unwrap` construct produces a `Result`, which is fully capable of representing any error `features.append_with_names` could produce.

> But that has nothing to do with the actual bug here, which was that the invariant failed.

The bug is by invoking `unwrap` the process crashed. To the degree that Cloudfare had a massive outage.

Had the logic been such that a `Result` representing this error condition activated an alternate workflow to handle the error (perhaps by logging it, emitting a notification event alerting SRE's, transitioning into a failure mode, or all of these options), then a global outage might have been averted.

Which makes:

> How they choose to implement checking and failing the invariant in the semantics of the chosen language is irrelevant.

Very relevant indeed.

Re: Cloudflare outage on November 18, 2025 post mortem

#943

Post by pkumar00007 »

If you knew the expected number of features , any input file with >100 should be discarded as bad input and you failback to the last good feature file received. This would have protected your service even though you are unable to get the newly populated features. I believe these features are not updated that frequently. Even if they were, you would have biased your system towards availability vs 'correctness'.

What were the teams doing between 11 to 1300 hrs , no explanation of what investigations were going on to not being able to figure the root cause.

Re: Cloudflare outage on November 18, 2025 post mortem

#944

Post by pkumar00007 »

Did your incident response team look at the last few changes that were executed? If they had , they could have just rolleback the change. or just looking at the changes executed, in the vicinity of the start of the outage could have pointed to the problem.

Didn't the services that were crashing due to OOM raise any alerts?

This is shitty at so many levels.

Re: Cloudflare outage on November 18, 2025 post mortem

#946

Post by gishh »

tucnak wrote: post #433
gishh wrote: post #61

Earlier quoted context omitted.

To be fair, if you’re not “this tall” you really shouldn’t consider using goto in a c program. Most people aren’t that tall.

Nonsense. Linux kernel for one example, uses goto everywhere for error handling.

How many codebases besides the Linux kernel had you seen an intentional amount of use of goto?

Re: Cloudflare outage on November 18, 2025 post mortem

#947

Post by mike_hearn »

bombela wrote: post #934
mike_hearn wrote: post #909

Earlier quoted context omitted.

There's clearly a big gap in how things are done in practice. You wouldn't see anyone call System.exit in a managed language if a data file was bigger than expected. You'd always get an exception. I used to be an SRE at Google. Back then we also had big outages caused by bad data files pushed to prod. It's a common enough issue so I really sympathize with Cloudflare, it's not nice to be on call for issues like that.…

A panic in Rust is the same as an exception in C++. You can catch it all the same. https://doc.rust-lang.org/std/panic/index.html An uncaught exception in C++ or an uncaught panic in Rust terminates the program. The unwinding is the same mechanism. I think the implementation is what comes with LLVM, but I haven't checked. I was also a Google SRE, and I liked the stacktrace facilities so much that I got permission to…

OK, so the issue is frameworks not catching panics and logging proper stack traces? Very cool that you made a library.

Re: Cloudflare outage on November 18, 2025 post mortem

#948

Post by proverbs53 »

What I read is that a non-critical feature (blocking / managing bot - access) was able to impact a critical feature (routing traffic).

Shouldn't the architecture setup in such a way that subcomponents can fail without impacting the critical function of the component?

Re: Cloudflare outage on November 18, 2025 post mortem

#950

Post by orphea »

gishh wrote: post #946
tucnak wrote: post #433

Earlier quoted context omitted.

Nonsense. Linux kernel for one example, uses goto everywhere for error handling.

How many codebases besides the Linux kernel had you seen an intentional amount of use of goto?

The answer is one search away on GitHub.

I will give you one example though: various .NET repositories (runtime, aspnetcore, orleans).

Post reply on HN