Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

721–730 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#721
post #642
post #582

Earlier quoted context omitted.

> I'm migrating my customers off Cloudflare. Is that an overreaction? Name me global, redundant systems that have not (yet) failed. And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic. I mean no service have 100% uptime - just that some have more nines than others.

Yes, it's probably an overreaction. But at the same time, what value do they add if they: * Took down the the customers sites due to their bug. * Never protected against an attack that our infra could not have handled by itself. * Don't think that they will be able to handle the "next big ddos" attack. It's just an extra layer of complexity for us. I'm sure there are attacks that could help our customers with, that's…

> • Took down the the customers sites due to their bug.

That is always a risk with using a 3rd party service, or even adding extra locally managed moving parts. We use them in DayJob, and despite this huge issue and the number of much smaller ones we've experienced over the last few years their reliability has been pretty darn good (at least as good as the Azure infrastructure we have their services sat in front of).

> • Never protected against an attack that our infra could not have handled by itself.

But what about the next one… Obviously this is a question sensitive to many factors in our risk profiles and attitudes to that risk, there is no one right answer to the “but is it worth it?” question here.

On a slightly facetious point: if something malicious does happen to your infrastructure, that it does not cope well with, you won't have the “everyone else is down too” shield :) [only slightly facetious because while some of our clients are asking for a full report including justification for continued use of CF and any other 3rd parties, which is their right both morally and as written in our contracts, most, especially those who had locally managed services affected, have taken the “yeah, half our other stuff was affected to, what can you do?” viewpoint].

> • Don't think that they will be able to handle the "next big ddos" attack.

It is a war of attrition. At some point a new technique, or just a new botnet significantly larger than those seen before, will come along that they might not be able to deflect quickly. I'd be concerned if they were conceited enough not to be concerned about that possibility. Any new player is likely to practise on smaller targets first before directly attacking CF (in fact I assume that it is rather rare that CF is attacked directly) or a large enough segment of their clients to cause them specific issues. Could your infrastructure do any better if you happen to be chosen as one of those earlier targets?

Again, I don't know your risk profile so can say which is the right answer, if there even is an easy one other than “not thinking about it at all” being a truly wrong answer. Also DDoS protection is not the only service many use CF for, so those need to be considered too if you aren't using them for that one thing.

Re: Cloudflare outage on November 18, 2025 post mortem

#722

The most surprising thing to me here is that it took 3 hours to root cause, and points to a glaring hole in the platform observability. Even taking into account the fact that the service was failing intermittently at first, it still took 1.5 hours after it started failing consistently to root cause. But the service was crashing on startup. If a core service is throwing a panic at startup like that, it should be raisi…

That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing with (a bot net that had escalated DDoS attacks from 3Tbps to 30Tbps against us and others like Microsoft over the last 3 months). We worked multiple theories in parallel. After an hour we ruled out the DDoS theory. We had other theories also running in parallel, but at that point the dominant theory was that the feature file was somehow corrupt. One thing that made us initially question the theory was nothing in our changelogs seemed like it would have caused the feature file to grow in size. It was only after the incident that we realized the database permissions change had caused it, but that was far from obvious. Even after we identified the problem with the feature file, we did not have an automated process to role the feature file back to a known-safe previous version. So we had to shut down the reissuance and manually insert a file into the queue. Figuring out how to do that took time and waking people up as there are lots of security safeguards in place to prevent an individual from easily doing that. We also needed to double check we wouldn’t make things worse. The propagation then takes some time especially because there are tiers of caching of the file that we had to clear. Finally we chose to restart the FL2 processes on all the machines that make up our fleet to ensure they all loaded the corrected file as quickly as possible. That’s a lot of processes on a lot of machines. So I think best description was it took us an hour for the team to coalesce on the feature file being the cause and then another two to get the fix rolled out.

Re: Cloudflare outage on November 18, 2025 post mortem

#723
post #595

Earlier quoted context omitted.

There are many self-hosted alternatives to protect against botnet. We don't have to use cloudflare. Everthing is under their control!

> There are many self-hosted alternatives to protect against botnet Whatever you do, unless you have their bandwidth capacity, at some point those "self-hosted" will get flooded with traffic.

As yourself more the question, is your service that important to need 99.999% uptime? Because i get the impression that people are so fixated on this uptime concept, that the idea of being down for a few hours is the most horrible issue in the world. To the point that they rather hand over control of their own system to a 3th party, then accept a downtime.

The fact that cloudflare can literally ready every bit of communication (as it sits between the client and your server) is already plenty bad. And yet, we accept this more easily, then a bit of downtime. We shall not ask about the prices for that service ;)

To me its nothing more then the whole "everybody on the cloud" issue, when most do not need the resource that cloud companies like AWS provide (and the bill), and yet, get totally tied down to this one service.

I am getting old lol ...

Re: Cloudflare outage on November 18, 2025 post mortem

#724

Earlier quoted context omitted.

This thread warms my heart. Rust has set a new baseline that many and myself now take for granted. We are now discussing what can be done to improve code correctness beyond memory and thread safety. I am excited for what is to come.

Alternatively you can look at actually innovative programming languages to peek at the next 20 years of innovation. I am not sure that watching the trendy forefront successfully reach the 1990s and discuss how unwrapping Option is potentially dangerous really warm my heart. I can’t wait for the complete meltdown when they discover effect systems in 2040. To be more serious, this kind of incident is yet another remind…

> I can’t wait for the complete meltdown when they discover effect systems in 2040

Zig is undergoing this meltdown. Shame it's not memory safe. You can only get so far in developing programming wisdom before Eternal September kicks in and we're back to re-learning all the lessons of history as punishment for the youthful hubris that plagues this profession.

Re: Cloudflare outage on November 18, 2025 post mortem

#725
post #546

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is the multi-million dollar .unwrap() story. While there are certainly many things to admire about Rust, this is why I prefer Golang's "noisy" error handling. In golang that would be either: feature_values, err := features.append_with_names(...) And the compiler would have complained that this value of `err` was unused; or you'd write: feature_values, _ := features.append_with_names(...) And it would be far mo…

> And it would be far more obvious that an error message is being ignored.

Haven't used Go so maybe I'm missing some consideration, but I don't see how ", _" is more obvious than ".unwrap()". If anything it seems less clear, since you need to check/know the function's signature to see that it's an error being ignored (wouldn't be the case for a function like https://pkg.go.dev/math#Modf).

Re: Cloudflare outage on November 18, 2025 post mortem

#726

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

If the error had been an exception instead of a result, could have bubbled up I have been saying for years that Rust botched error handling in unfixable ways. I will go to the grave believing Rust fumbled. The design of the Rust language encourages people to use unwrap() to turn foreseeable runtime problems into fatal errors. It's the path of least resistance, so people will take it. Rust encourages developers to con…

Errors work just like exceptions especially if you use the ? operator and let the error bubble up the chain. This is the Rust equivalent of an unhandled exception and the ripcord being pulled.

Re: Cloudflare outage on November 18, 2025 post mortem

#727

Earlier quoted context omitted.

> at least do `expect("damn it, how did this happen")` That gives you the same behavior as unwrap with a less useful error message though. In theory you can write useful messages, but in practice (and your example) expect is rarely better than unwrap in modern rust

We shouldn't be using unwrap() or expect() at all. This is Rust's Null Pointer Exception. unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features. The language should grow the ability to mark this code as dangerous, and we should have static tools to exclude this code from our dependency tree. I don't want some library I use to `u…

I'm on the Rust libs-api team and you're mistaken. I use `unwrap()` all the time.

My blog on this topic was linked above, you should read it: https://burntsushi.net/unwrap/

Re: Cloudflare outage on November 18, 2025 post mortem

#728
post #653

Earlier quoted context omitted.

I don't think these are realistic requirements for any engineered system to be honest. Realistic is to have contingencies for such cases, which are simply errors. But the case for Cloudflare here is complicated. Every engineer is very free to make a better system though.

What is not realistic? To do simple input validation on data that has the potential to break 20% of the internet? To not have a system in place to rollback to the latest known state when things crash? Cloudflare builds a global scale system, not an iphone app. Please act like it.

Cloudflares success was simplicity to build a distributed system in different data centers around the world to be implemented by third party IT workers while Cloudflare were a few people. There are probably a lot of shitty iPhone apps that do less important work and are vastly more complex than the former Cloudflare server node configuration.

Every system has a non-reducible risk and no data rollback is trivial, especially for a CDN.

Re: Cloudflare outage on November 18, 2025 post mortem

#729
post #495

Earlier quoted context omitted.

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality. Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled. If you need the value, you need to handle the case that it’s not available explicitly. Y…

The blog linked in the GP anticipates this rebuttal and already addresses it.

Your argument also implies that things like `slice[i]` are never okay.

Re: Cloudflare outage on November 18, 2025 post mortem

#730

Earlier quoted context omitted.

Yeah even if you handled this situation without unwrap() if you just went down an error path that didn't panic, the service would likely still be inoperable if every single request went down the error path.

Okay, but if you returned a wrapped error it’d at least be easier to debug.

not necessarily, by default `Result` does not even carry a stack trace
Post reply on HN