Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

701–710 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#701
post #654
post #501

Earlier quoted context omitted.

It doesn’t disappear, it forces you to handle it.

Propagating upwards a valid way of handling it and often the correct answer. There needs to be something at the top level that can handle a crashing process.

can Rust handle global panics?

Or can a unwrap be stopped?

This is just a normal Tuesday for languages with Exception and try/catch.

Re: Cloudflare outage on November 18, 2025 post mortem

#702

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

I would say that whilst this is a good top down view, that `.unwrap()` should have been caught at code-review and not allowed. Clippy rule could have saved a lot of money. That and why the hell wasn't their alerting showing up colossal amount of panics in their bot manager thing?

Yes the lack of observability is really the disturbing bit here. You have panics in a bunch of your core infrastructure, you would expect there to be a big red banner on the dashboard that people look at when they first start troubleshooting an incident.

This is also a pretty good example why having stack traces by default is great. That error could have been immediately understood just from a stack trace and a basic exception message.

Re: Cloudflare outage on November 18, 2025 post mortem

#703

Earlier quoted context omitted.

I would be fine just getting rid of unwrap(), expect(), etc. That's still a net win. Look at how many lazy cases of this there are in Rust code [1]. Some of these are no doubt tested (albeit impossible to statically guarantee), but a lot of it looks like sloppiness or not leaning on the language's strong error handling features. It's disappointing to see. We've had so much of this creep into the language that eventua…

I don't write Rust so I don't really know, but from someone else's description here it sounds similar to `fromJust` in Haskell which is a common newbie footgun. I think you're right that this is a case of not using the language properly, though I know I was seduced into the idea that Haskell is safe by default when I was first learning, which isn't quite true — the safety features are opt-in. A language DX feature I…

I would be in favor of renaming unwrap() and its family to `unwrap_do_not_use_or_you_will_break_the_internet()`

I still think we should remove them outright or make production code fail to compile without a flag allowing them. And we also need tools to start cleaning up our dependency tree of this mess.

Re: Cloudflare outage on November 18, 2025 post mortem

#704

Earlier quoted context omitted.

Except it fails to document anything about the actions they made to Warp in London during the resolution.

There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wante…

Thanks!

Re: Cloudflare outage on November 18, 2025 post mortem

#705

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Oh come on, stop spreading FUD. Rust programs are 100% immune to crashes and bugs, they have memory safety (c).

Also, exception handling is hard and lame. We don't need exceptions, just add a "match" block after every line in your program.

Re: Cloudflare outage on November 18, 2025 post mortem

#706
post #653

Earlier quoted context omitted.

I don't think these are realistic requirements for any engineered system to be honest. Realistic is to have contingencies for such cases, which are simply errors. But the case for Cloudflare here is complicated. Every engineer is very free to make a better system though.

What is not realistic? To do simple input validation on data that has the potential to break 20% of the internet? To not have a system in place to rollback to the latest known state when things crash? Cloudflare builds a global scale system, not an iphone app. Please act like it.

Yeah, I don't quite understand the people cutting Cloudflare massive slack. It's not about nailing blame on a single person or a team, it's about keeping a company that is THE closest thing to a public utility for the web accountable. They more or less did a Press Release with a call to action to buy or use their services at the end and everybody is going "Yep, that's totally fine. Who hasn't sent a bug to prod, amirite?".

It goes over my head why Cloudflare is HN's darling while others like Google, Microsoft and AWS don't usually enjoy the same treatment.

Re: Cloudflare outage on November 18, 2025 post mortem

#707
post #406

Earlier quoted context omitted.

> You misunderstand what Rust’s guarantees are. Well, no, most Rust programmers misunderstand what the guarantees are because they keep parroting this quote. Obviously the language does not protect you from logic errors, so saying "if it compiles, it works" is disingenuous, when really what they mean is "if it compiles, it's probably free of memory errors".

No, the "if it compiles, it works" is genuinely about the program being correct rather than just free of memory errors, but it's more of a hyperbolic statement than a statement of fact. It's a common thing I've experienced and seen a lot of others say that the stricter the language is in what it accepts the more likely it is to be correct by the time you get it to run. It's not just a Rust thing (although I think Rus…

Everyone understands Rust doesn't offer such guarantees.

Even more now after this outage.

But it's a fact that "if it compiles it runs" is often associated with Rust, in HN at least. A quick Algolia search tells me that.

Re: Cloudflare outage on November 18, 2025 post mortem

#708

Earlier quoted context omitted.

You don’t even need all the ceremony. If the config gets updated every 5 minutes, it surely is being hot-reloaded. If that’s the case, the old config is already in memory when the new config is being parsed. If that’s the case, parsing shouldn’t have panicked, but logged a warning, and carried on with the old config that must already be in memory.

> If that’s the case, the old config is already in memory when the new config is being parsed I think that's explicitly a non-goal. My understanding is that Cloudflare prefers fail safe (blocking legitimate traffic) over fail open (allowing harmful traffic).

Well, they should then add some reliability goals into the mix too to balance it out a bit.

Re: Cloudflare outage on November 18, 2025 post mortem

#709
I’ll be honest, I only understand about 30% of what is being said in this thread and that is probably generous. But it is very interesting seeing so many people respond to each other “it’s so simple! what went wrong was…” as they all disagree on what exactly went wrong.

Re: Cloudflare outage on November 18, 2025 post mortem

#710

Earlier quoted context omitted.

Well let me avoid those that don’t understand it. It’s literally Rust 101.

It's literally not, Rust tutorials are littered with `.unwrap()` calls. It might be Rust 102, but the first impression given is that the language is surprisingly happy with it.

https://doc.rust-lang.org/book/ch09-02-recoverable-errors-wi...

If you haven't read the Rust Book at least, which is effectively Rust 101, you should not be writing Rust professionally. It has a chapter explaining all of this.

Post reply on HN