Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

751–760 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#751

Post by underdeserver »

burntsushi wrote: post #727
echelon wrote: post #636

Earlier quoted context omitted.

We shouldn't be using unwrap() or expect() at all. This is Rust's Null Pointer Exception. unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features. The language should grow the ability to mark this code as dangerous, and we should have static tools to exclude this code from our dependency tree. I don't want some library I use to `u…

I'm on the Rust libs-api team and you're mistaken. I use `unwrap()` all the time. My blog on this topic was linked above, you should read it: https://burntsushi.net/unwrap/

Eh, "mistaken" might be a bit harsh. He's stating an opinion, which you and I disagree with.

> The language should grow the ability to mark this code as dangerous, and we should have static tools to exclude this code from our dependency tree.

Might be useful to point out that this static tool exists (clippy::unwrap_used).

Re: Cloudflare outage on November 18, 2025 post mortem

#752

Post by miki123211 »

huijzer wrote: post #615
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> They failed on so many levels here. That's often the case with human error as especially aviation safety experts know: https://en.wikipedia.org/wiki/Swiss_cheese_model

In organizations with this level of care, if you fail at fewer levels, customers just never notice the error.

Any big and noticeable incident is one of the "we failed on so many levels here" kind, by definition.

Re: Cloudflare outage on November 18, 2025 post mortem

#753

Post by mike_hearn »

dehrmann wrote: post #456
gwbas1c wrote: post #281

Earlier quoted context omitted.

It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.) Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is e…

> a stack trace takes a lot of overhead to generate Can't Hotspot not generate the stack trace when it knows the exception will be caught and the stack trace ignored?

It can and that optimization has existed for a while.

Actually it can also just turn off the collection of stack traces entirely for throw sites that are being hit all the time. But most Java code doesn't need this because code only throws exceptions for exceptional situations.

Re: Cloudflare outage on November 18, 2025 post mortem

#755

Post by ziml77 »

dev_l1x_be wrote: post #488
echelon wrote: post #311

Earlier quoted context omitted.

A lot of stuff should be done about the awful unwrap family of methods. A few ideas: - It should not compile in production Rust code - It should only be usable within unsafe blocks - It should require explicit "safe" annotation from the engineer. Though this is subject to drift and become erroneous. - It should be possible to ban the use of unsafe in dependencies and transitive dependencies within Cargo.

Than they are going to write None | Err => yolo() that has the same impact. It is not the syntax or the semantic meaning is the problem here but the fact that there is no monitoring around the elevated error counts after a deployment. Software engineers tend to get stuck in software problems and thinking that everything should be fixed in code. In reality there are many things outside of the code that you can do to o…

Exactly. People are very hung up on "unwrap" but even if it wasn't there at all, you will have devs just manually writing the match. Or, even more likely, using a trivial 'unwrap!" macro.

There's also an assumption here that if the unwrap wasn't there, the caller would have handled the error properly. But if this isn't part of some common library at CF, then chances are the caller is the same person who wrote the panicking function in the first place. So if a new error variant they introduced was returned they'd probably still abort the thread either by panicking at that point or breaking out of the thread's processing loop.

Re: Cloudflare outage on November 18, 2025 post mortem

#756

Post by grogers »

torginus wrote: post #745
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

By the way - does this discussion matter and were they wrong to use unwrap()? The way they wrote the code means that having more than 200 features is a hard non-transient error - even if they recovered from it, it meant they'd have had the same error when the code got to the same place. I'm sure when the process crashed, k8s restarted the pod or something - then it reran the same piece of code and crashed in the same…

Instead of crashing when applying the new config, it's more common to simply ignore the new config if it cannot be applied. You keep running in the last known good state. Operators then get alerts about the failures and can diagnose and resolve the underlying issue.

That's not always foolproof, e.g. a freshly (re)started process doesn't have any prior state it can fall back to, so it just hard crashes. But restarts are going to be rate limited anyways, so even then there is time to mitigate the issue before it becomes a large scale outage

Re: Cloudflare outage on November 18, 2025 post mortem

#757

Post by lgeek »

monerozcash wrote: post #657
purple_turtle wrote: post #608

Earlier quoted context omitted.

Can you name three of this many alternatives? How they magically manage DDOS larger than their bandwidth? If the plan is to have larger bandwidth than any DDOS it is going to be expensive, quickly.

You could probably get a very fat pipe with usage based billing, you'd only go bankrupt when you get hit by a big DDoS and not before.

If you're buying transit, you'll have a hard time getting away with less than 10% commit, i.e. you'll have to pay for 10 Gbps of transit to have a 100 Gbps port, which will typically run into 4 digits USD / month. You'll need a few hundred Gbps of network and scrubbing capacity to handle common DDoS attacks using amplification from script kids with a 10 Gbps uplink server that allow spoofing, and probably on the order of 50+ Tbps to handle Aisuru.

If you're just renting servers instead, you have a few options that are effectively closer to a 1% commit, but better have a plan B for when your upstreams drop you if the incoming attack traffic starts disrupting other customers - see Neoprotect having to shut down their service last month.

Re: Cloudflare outage on November 18, 2025 post mortem

#758

Post by globalise83 »

tetec1 wrote: post #59
eastdakota wrote: post #35

Earlier quoted context omitted.

Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.

Yeah, I can imagine that this insertion was some high-pressure job.

The computer science equivalent of choosing between the red, green and blue wires when disarming a nuke with 15 seconds left on the clock

Re: Cloudflare outage on November 18, 2025 post mortem

#759

Post by jamesblonde »

Cloudflare tried to build their own feature store, and get a grade F.

I wrote a book on feature stores by O'Reilly. The bad query they wrote in Clickhouse could have been caused by another more error - duplicate rows in materialized feature data. For example, in Hopsworks it prevents duplicate rows by building on primary key uniqueness enforcement in Apache Hudi. In contrast, Delta lake and Iceberg do not enforce primary key constraints, and neither does Clickhouse. So they could have the same bug again due to a bug in feature ingestion - and given they hacked together their feature store, it is not beyond the bounds of possibility.

Reference: https://www.oreilly.com/library/view/building-machine-learni...

Re: Cloudflare outage on November 18, 2025 post mortem

#760

Post by mike_hearn »

bombela wrote: post #490
wrs wrote: post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

This thread warms my heart. Rust has set a new baseline that many and myself now take for granted. We are now discussing what can be done to improve code correctness beyond memory and thread safety. I am excited for what is to come.

Really not! This is a huge faceplant for writing things in Rust. If they had been writing their code in Java/Kotlin instead of Rust, this outage either wouldn't have happened at all (a failure to load a new config would have been caught by a defensive exception handler), or would have been resolved in minutes instead of hours.

The most useful thing exceptions give you is not static compile time checking, it's the stack trace, error message, causal chain and ability to catch errors at the right level of abstraction. Rust's panics give you none of that.

Look at the error message Cloudflare's engineers were faced with:

     thread fl2_worker_thread panicked: called Result::unwrap() on an Err value
That's useless, barely better than "segmentation fault". No wonder it took so long to track down what was happening.

A proxy stack written in a managed language with exceptions would have given an error message like this:

    com.cloudflare.proxy.botfeatures.TooManyFeaturesException: 200 > 60
        at com.cloudflare.proxy.botfeatures.FeatureLoader(FeatureLoader.java:123)
        at ...
and so on. It'd have been immediately apparent what went wrong. The bad configs could have been rolled back in minutes instead of hours.

In the past I've been able to diagnose production problems based on stack traces so many times I was been expecting an outage like this ever since the trend away from providing exceptions in new languages in the 2010s. A decade ago I wrote a defense of the feature and I hope we can now have a proper discussion about adding exceptions back to languages that need them (primarily Go and Rust):

https://blog.plan99.net/what-s-wrong-with-exceptions-nothing...

Post reply on HN