Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

821–830 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#821

Post by mamcx »

wrs wrote: post #86
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

The problem here is that for a service "fail fast" is mostly the best default.

Re: Cloudflare outage on November 18, 2025 post mortem

#822

Post by asa400 »

tptacek wrote: post #167
otterley wrote: post #160

Earlier quoted context omitted.

It's great that you're working on regionalization. Yes, it is hard, but 100x harder if you don't start with cellular design in mind. And as I said in the root of the thread, this is a sign that CloudFlare needs to invest in it just like you have been.

I recoil from that last statement not because I have a rooting interest in Cloudflare but because the last several years of working at Fly.io have drilled Richard Cook's "How Complex Systems Fail"† deep into my brain, and what you said runs aground of Cook #18: Failure free operations require experience with failure. If the exact same thing happens again at Cloudflare, they'll be fair game. But right now I feel peopl…

Thank you for saying it. I’m getting exasperated at how many people in the comments are making some variant of the “lazy programmer wrote code that took a shortcut” argument.

Complex system failures are not monocausal! Complex systems are in a continuous state of partial failure!

Re: Cloudflare outage on November 18, 2025 post mortem

#823

Post by empath75 »

gwd wrote: post #561
brabel wrote: post #495

Earlier quoted context omitted.

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

> Yes, I always thought it was wrong to use unwrap in examples. And because it gets picked up by LLMs. It would be interesting to know if this particular .unwrap() was written by a human.

I have removed many 'unwraps' from copilot and cursor generated code, but cursor is a lot better at it than copilot was.

Re: Cloudflare outage on November 18, 2025 post mortem

#824

Post by aquariusDue »

miyuru wrote: post #741
aquariusDue wrote: post #706

Earlier quoted context omitted.

Yeah, I don't quite understand the people cutting Cloudflare massive slack. It's not about nailing blame on a single person or a team, it's about keeping a company that is THE closest thing to a public utility for the web accountable. They more or less did a Press Release with a call to action to buy or use their services at the end and everybody is going "Yep, that's totally fine. Who hasn't sent a bug to prod, amir…

>It goes over my head why Cloudflare is HN's darling while others like Google, Microsoft and AWS don't usually enjoy the same treatment. Do the others you mentioned provide such detailed outage reports, within 24 hours of an incident? I’ve never seen others share the actual code that related to the incident. Or the CEO or CTO replying to comments here? >Press Release This is not press release, they always did these o…

> Do the others you mentioned provide such detailed outage reports, within 24 hours of an incident? I’ve never seen others share the actual code that related to the incident.

Azure (albeit pretty old): https://devblogs.microsoft.com/devopsservice/?p=17665

AWS: https://aws.amazon.com/message/101925/

GCP: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

The code sample might as well be COBOL for people not familiar with Rust and its error handling semantics.

> Or the CEO or CTO replying to comments here?

I've looked around the thread and I haven't seen the CTO here nor the CEO, probably I'm not familiar with their usernames and that's on me.

> This is not press release, they always did these outage posts from the start of the company.

My mistake calling them press releases. Newspapers and online publications also skim this outage report to inform their news stories.

I wasn't clear enough on my previous comment. I'd like all major players in the internet and web infrastructure to be held to higher standards. As it stands when it comes to them or the tech department of a retail store the retail store must answer to more laws when surface area of combined activities is took into account.

Yes, Cloudflare excels where others don't or barely bother and I too enjoyed the pretty graphs, diagrams and I've learned some nifty Rust tricks.

EDIT: I've removed some unwarranted snark from my comment which I apologize for.

Re: Cloudflare outage on November 18, 2025 post mortem

#825

Post by tptacek »

JB_Dev wrote: post #447
tptacek wrote: post #17

Earlier quoted context omitted.

I don't think this system is best thought of as "deployment" in the sense of CI/CD; it's a control channel for a distributed bot detection system that (apparently) happens to be actuated by published config files (it has a consul-template vibe to it, though I don't know if that's what it is).

Code and Config should be treated similarly. If you would use a ring based rollout, canaries, etc for safely changing your code, then any config that can have the same impact must also use safe rollout techniques.

You're the nth person on this thread to say that and it doesn't make sense. Events that happen multiple times per second change data that you would call "configuration" in systems like these. This isn't `sendmail.cf`.

If you want to say that systems that light up hundreds of customers, or propagate new reactive bot rules, or notify a routing system that a service has gone down are intrinsically too complicated, that's one thing. By all means: "don't build modern systems! computers are garbage!". I have that sticker on my laptop already.

But like: handling these problems is basically the premise of large-scale cloud services. You can't just define it away.

Re: Cloudflare outage on November 18, 2025 post mortem

#826

Post by empath75 »

quotemstr wrote: post #711
frumplestlatz wrote: post #549

Earlier quoted context omitted.

I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality. Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled. If you need the value, you need to handle the case that it’s not available explicitly. Y…

> If you have to use unwrap, your types do not match your problem The problem starts with Rust stdlib. It panics on allocation failure. You expect Rust programmers to look at stdlib and not imitate it? Sure, you can try to taboo unwrap(), but 1) it won't work, and 2) it'll contort program design in places where failure really is a logic bug, not a runtime failure, and for which unwrap() is actually appropriate. The r…

We flag unwrap/expect usage in lints and have limited it to just server startup where we want the server to crash if a file is missing...

Re: Cloudflare outage on November 18, 2025 post mortem

#827

Post by empath75 »

jandrewrogers wrote: post #737
inferiorhuman wrote: post #521

Earlier quoted context omitted.

Dunno, I think the alternatives have their own pretty significant downsides. All would require front loading more in-depth understanding of error handling and some would just be quite a bit more verbose. IMO making unwrap a clippy lint (or perhaps a warning) would be a decent start. Or maybe renaming unwrap.

This strikes me as a culture issue more than one of language. A tenet of systems code is that every possible error must be handled explicitly and exhaustively close to the point of occurrence. It doesn’t matter if it is Rust, C, etc. Knowing how to write systems code is unrelated to knowing a systems language. Rust is a systems language but most people coming into Rust have no systems code experience and are “holding…

Unwrap _is_ explicitly handling an error at the point of occurrence. You have explicitly decided to panic, which is sometimes a valid choice. I use it (on startup only) when server configs are missing or invalid or in CLI tools when the options aren't valid. Crashing a pod on startup before it goes Ready is a valid pattern in k8s and generally won't cause an outage because the previous pod will continue working.

Re: Cloudflare outage on November 18, 2025 post mortem

#828

Post by nijave »

thatoneengineer wrote: post #413

The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen". The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they w…

Quite surprising a single bad config file brought down their entire global network across multiple products

Re: Cloudflare outage on November 18, 2025 post mortem

#829

Post by gwbas1c »

branko_d wrote: post #469
gwbas1c wrote: post #281

Earlier quoted context omitted.

It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.) Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is e…

> Exceptions force a panic on all errors What do you mean? Exceptions do not force panic at all. In most practical situations, an exception unhandled close to where it was thrown will eventually get logged. It's kind of a "local" panic, if you will, that will terminate the specific function, but the rest of the program will remain unaffected. For example, a web server might throw an exception while processing a speci…

> Throwing an exception does not necessarily mean that your program is suddenly in an unsupported state, and therefore does not require terminating the entire program.

That's not what a panic means. Take a read through Go's panic / resume mechanism; it's similar to exceptions, but the semantics (with multiple return values) make it clear that panic is for exceptional situations. (IE, panic isn't for "file not found," but instead it's for when code isn't written to handle "file not found.")

Even Rust has mechanisms to panic without aborting the process, although I will readily admit that I haven't used them and don't understand them: https://doc.rust-lang.org/std/panic/fn.resume_unwind.html

Re: Cloudflare outage on November 18, 2025 post mortem

#830

Post by nijave »

jve wrote: post #582
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> I'm migrating my customers off Cloudflare. Is that an overreaction? Name me global, redundant systems that have not (yet) failed. And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic. I mean no service have 100% uptime - just that some have more nines than others.

We had better uptime with AWS WAF in us-east-1 than we've had in the last 1.5 years of Cloudflare.

I do like the flat cost of Cloudflare and feature set better but they have quite a few outages compared to other large vendors--especially with Access (their zero trust product)

I'd lump them into GitHub levels of reliability

We had a comparable but slightly higher quote from an Akamai VAR.

Post reply on HN