Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

871–880 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#871

Post by frumplestlatz »

burntsushi wrote: post #788
frumplestlatz wrote: post #765

Earlier quoted context omitted.

`slice[i]` is also a hole in the type system, but at least it’s generally relying on a local invariant, immediate to the surrounding context, that does not require lying about invariants across your API surface. The blog post doesn’t address the issue, it simply pretends it’s not a real problem. Also from the post: “If we were to steelman advocates in favor of this style of coding, then I think the argument is probab…

`slice[i]` is just sugar for `slice.get(i).unwrap()`. And whether it's a "local" invariant or not is orthogonal. And `unwrap()` does not "require lying about invariants across your API surface." > The blog post doesn’t address the issue, it simply pretends it’s not a real problem. It very explicitly addresses it! It even gives real examples. > Also from the post: “If we were to steelman advocates in favor of this sty…

> `slice[i]` is just sugar for `slice.get(i).unwrap()`. And whether it's a "local" invariant or not is orthogonal. And `unwrap()` does not "require lying about invariants across your API surface."

It's not orthogonal. `Result` isn't a local invariant, and yes, `.unwrap()` does require lying. If your code depends on an API that can fail, and you cannot handle that failure locally (`.unwrap()` is not handling it), then your type signature needs to express that you can fail -- and you need to raise an error on that failure.

> That's an extreme position. It isn't caveated to only apply to certain contexts.

No, it's a principled position. Correct code doesn't `.unwrap()`, but code that hides failure cases -- or foists invariant enforcement onto programmers remembering not to screw up -- does.

I've built and worked on ridiculously complex code bases without a single instance of `.unwrap()` or the local language equivalent; it's just not necessary. This is just liked the unchecked exception debate in Java -- complex explanations for a very simple goal of avoiding the thought, time, and effort to accurately model a system's invariants.

Re: Cloudflare outage on November 18, 2025 post mortem

#872

Post by Diggsey »

There were two things I think went extremely poorly here:

1) Lack of validation of the configuration file.

Rolling out a config file across the global network every 5 minutes is extremely high risk. Even without hindsight, surely one would see then need for very careful validation of this file before taking on that risk?

There were several things "obviously" wrong with the file that validation should have caught:

- It was much bigger than expected.

- It had duplicate entries.

- Most importantly, when loaded into the FL2 proxy, the proxy would panic on every request. At the very least, part of the validation should involve loading the file into the proxy and serving a request?

2) Very long time to identify and then fix such a critical issue.

I can't understand the complete lack of monitoring or reporting? A panic in Rust code, especially from an unwrap, is the application screaming that there's a logic error! I don't understand how that can be conflated with a DDoS attack. How are your logs not filled with backtraces pointing to the exact "unwrap" in question?

Then, once identified, why was it so hard to revert to a known good version of the configuration file? How did noone foresee the need to roll back this file when designing a feature that deploys a new one globally every 5 minutes?

Re: Cloudflare outage on November 18, 2025 post mortem

#873

Post by MaxBarraclough »

nomel wrote: post #178
yoyohello13 wrote: post #163

Earlier quoted context omitted.

Probably not, since errors as values are way better than exceptions.

How so? An exception is a value that's given the closest, conceptually appropriate, point that was decided to handle the value, allowing you to keep your "happy path" as clean code, and your "exceptional circumstances" path at the level of abstraction that makes sense. It's way less book-keeping with exceptions, since you, intentionally, don't have to write code for that exceptional behavior, except where it makes se…

> don't have to write code for that exceptional behavior, except where it makes sense to.

The great Raymond Chen wrote an excellent blog post on how this isn't really true, and how exceptions can lure programmers into mistakenly thinking they can just forget about failure cases.

Cleaner, more elegant, and harder to recognize https://devblogs.microsoft.com/oldnewthing/20050114-00/?p=36...

(ctrl-f for taskbar to skip to heart of his point.)

Re: Cloudflare outage on November 18, 2025 post mortem

#874

Post by aforwardslash »

polack wrote: post #570
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

I know its easy to criticize what happened after the fact and having a clear(er) picture of all the moving parts and the timeline of events, but I think that while most of the people in the thread are pointing out either Rust-related or lack of configuration validation, what really grinds my gears is something that - in my opinion - is bad engineering.

Having an unprivileged application querying system.columns to infer the table layout is just bad; Not having a proper, well-defined table structure indicates sloppiness in the overall schema design, specially if it changes quickly. Considering specifically clickhouse, and even if this approach would be a good idea, the unprivileged way of doing it would be "DESCRIBE TABLE ", NOT iterating system.columns. The gist of it - sloppy design not even well implemented.

Having a critical application issuing ad-hoc commands to system.* tablespace instead of using a well-tested library is just amateurism, and again - bad engineering; IMO it is good practice to consider all system.* privileged applications and ensure their querying is completely separate from your application logic; Sometimes some system tables change, and fields are added and/or removed - not planning for this will basically make future compatibility a nightmare.

Not only the problematic query itself, but the whole context of this screams "lack of proper application design" and devs not knowing how to use the product and/or read the documentation. Granted, this is a bit "close to home" for me, because I use ClickHouse extensively (at a scale - I'm assuming - several orders of magnitude smaller than CloudFlare) and I have spent a lot of time designing specifically to avoid at least some of these kind of mistakes. But, if I can do it at my scale, why aren't they doing it?

Re: Cloudflare outage on November 18, 2025 post mortem

#875

Post by EvanAnderson »

tptacek wrote: post #825
JB_Dev wrote: post #447

Earlier quoted context omitted.

Code and Config should be treated similarly. If you would use a ring based rollout, canaries, etc for safely changing your code, then any config that can have the same impact must also use safe rollout techniques.

You're the nth person on this thread to say that and it doesn't make sense. Events that happen multiple times per second change data that you would call "configuration" in systems like these. This isn't `sendmail.cf`. If you want to say that systems that light up hundreds of customers, or propagate new reactive bot rules, or notify a routing system that a service has gone down are intrinsically too complicated, that'…

I'm sorry to belabor this but I'm genuinely not understanding what you're saying in this reply. I haven't operated large scale systems. I'm just an IT generalist and casual coder. I acknowledge I'm too inexperienced to even know what I don't know re: running large systems.

I read the parent poster as broadly suggesting configuration updates should have fitness tests applied and be deployed to minimize the blast radius when an update causes a malfunction. That makes intuitive sense to me. It seems like software should be subject to health checks after configuration updates, even if it's just to stop a deployment before it's widely distributed (let alone rolling-back to last-working configurations, etc).

Am I being thick-headed in thinking defensive strategies like those are a good idea? I'm reading your reply as arguing against those types of strategies. I'm also not understanding what you're suggesting as an alternative.

Again, I'm sorry to belabor this. I've replied once, deleted it, tried writing this a couple more times and given up, and now I'm finally pulling the trigger. It's really eating at me. I feel as though I must be deep down the Dunning-Kruger rabbit hole and really thinking "outside my lane".

Re: Cloudflare outage on November 18, 2025 post mortem

#876

Post by bungle »

vbezhenar wrote: post #607
thatoneengineer wrote: post #413

The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen". The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they w…

IMO: there should be explicit error path for invalid configuration, so the program would abort with specific exit code and/or message. And there should be a superviser which would detect this behaviour, rollback old working config and wait for few minutes before trying to apply new config again (of course with corresponding alerts). So basically bad config should be explicitly processed and handled by rolling back to…

System outputting the configuration file failed (it could check the size and/or content and stop right away), but also a system importing the file failed. These usually sound simple/stupid in a hindsight. I am not a fan of everything centralising to a few hands. As in bad situation, they can also be weaponised or attacked. And in good situation their blast radius is just too big and a bit random, in this case global.

Re: Cloudflare outage on November 18, 2025 post mortem

#877

Post by samdoesnothing »

yoyohello13 wrote: post #831
samdoesnothing wrote: post #489

Earlier quoted context omitted.

> but Rust's type system did catch this error - and then author decided it's fine to panic if this error happens The issue is that it wasn't fine to panic, thus Rust did not catch this error.

Come on now. You can't blame the compiler when the programmer explicitly told the compiler to not worry about it. There is nothing in existence that can protect against something like that.

I'm not blaming the compiler, of course programmers can explicitly write incorrect code in any language. It's not Rust's fault.

Re: Cloudflare outage on November 18, 2025 post mortem

#878

Post by tptacek »

EvanAnderson wrote: post #875
tptacek wrote: post #825

Earlier quoted context omitted.

You're the nth person on this thread to say that and it doesn't make sense. Events that happen multiple times per second change data that you would call "configuration" in systems like these. This isn't `sendmail.cf`. If you want to say that systems that light up hundreds of customers, or propagate new reactive bot rules, or notify a routing system that a service has gone down are intrinsically too complicated, that'…

I'm sorry to belabor this but I'm genuinely not understanding what you're saying in this reply. I haven't operated large scale systems. I'm just an IT generalist and casual coder. I acknowledge I'm too inexperienced to even know what I don't know re: running large systems. I read the parent poster as broadly suggesting configuration updates should have fitness tests applied and be deployed to minimize the blast radiu…

The things you do to safeguard the rollout of a configuration file change are not the same as the things you do to reliably propagate changes that might happen many times per second.

What's irritating to me are the claims that there's nothing distinguishing real time control plane state changes and config files. Most of us have an intuition for how they'd do a careful rollout of a config file change. That intuition doesn't hold for control plane state; it's like saying, for instance, that OSPF should have canaries and staged rollouts every time a link state changes.

I'm not saying there aren't things you to do make real-time control plane state propagation safer, or that Cloudflare did all those things (I have no idea, I'm not familiar with their system at all, which is another thing irritating me about this thread --- the confident diagnostics and recommendations). I'm saying that people trying to do the "this is just like CrowdStrike" thing are telling on themselves.

Re: Cloudflare outage on November 18, 2025 post mortem

#879

Post by ok123456 »

echelon wrote: post #655
milliams wrote: post #600

Earlier quoted context omitted.

[lints.clippy] dbg_macro = "deny" unwrap_used = "deny" expect_used = "deny"

https://github.com/search?q=unwrap%28%29+language%3ARust&typ... This is sobering. My new fear is some dependency unwrap()ing or expect()ing something where they didn't prove the correctness. Unwrap() and expect() are an anti-pattern and have no place in idiomatic Rust code. The language should move to deprecate them.

Unwrap is in a lot of example code.

If you're using Result, there's no automatic language feature for statically typing a nested E that mirrors how it was called.

So out of brevity, they unwrap.

Expect to see this sort of error crop up a lot as people use LLMs to vibe with the borrow checker.

Re: Cloudflare outage on November 18, 2025 post mortem

#880

Post by richardwhiuk »

karel-3d wrote: post #516
richardwhiuk wrote: post #470

Earlier quoted context omitted.

Because Go silently gives you zero/null instead

Idiomatically, it gives you `err` and you do `if err != nil {return err}`. While in rust you mostly do `.unwrap` and panic. It's not in the type system, but it's idiomatic

Get a key from a map and forget to check the error.
Post reply on HN