Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

901–910 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#901

Post by frumplestlatz »

waterTanuki wrote: post #64
vsgherzi wrote: post #22

Why does cloudflare allow unwraps in their code? I would've assumed they'd have clippy lints stopping that sort of thing. Why not just match with { ok(value) => {}, Err(error) => {} } the function already has a Result type. At the bare minimum they could've used an expect("this should never happen, if it does database schema is incorrect"). The whole point of errors as values is preventing this kind of thing.... It w…

Not a cloudflare employee but I do write a lot of Rust. The amount of things that can go wrong with any code that needs to make a network call is staggeringly high. unwrap() is normal during development phase but there are a number of times I leave an expect() for production because sometimes there's no way to move forward.

At risk of sounding harsh, that’s a huge failure in your modeling of invariants that should not be permitted in development.

Permitting it in development is why one ends up in the position of having to use an `expect()` in production code, because your API surfaces are wrong and can’t model your actual invariants.

Re: Cloudflare outage on November 18, 2025 post mortem

#902

Post by frumplestlatz »

hu3 wrote: post #701
VMG wrote: post #654

Earlier quoted context omitted.

Propagating upwards a valid way of handling it and often the correct answer. There needs to be something at the top level that can handle a crashing process.

can Rust handle global panics? Or can a unwrap be stopped? This is just a normal Tuesday for languages with Exception and try/catch.

> This is just a normal Tuesday for languages with Exception and try/catch.

Yes, unfortunately, random stack unrolls and weird state bugs as a result are a normal Tuesday for languages with (unchecked) Exception and try/catch

Re: Cloudflare outage on November 18, 2025 post mortem

#903

Post by frumplestlatz »

rixed wrote: post #463
RagingCactus wrote: post #102

Lots of people here are (perhaps rightfully) pointing to the unwrap() call being an issue. That might be true, but to me the fact that a reasonably "clean" panic at a defined line of code was not quickly picked up in any error monitoring system sounds just as important to investigate. Assuming something similar to Sentry would be in use, it should clearly pick up the many process crashes that start occurring right as…

Exactly! You could have `rand() > 0.5 && panic!()` in the code of your bot module, and that should not put the internet on fire. The issue here is about the system as a whole not any line of code.

> The issue here is about the system as a whole not any line of code.

Unsoundness in the type system that leads to a systemic failure is about the system as a whole.

Not everything can be recovered from restarting a process, and process correctness and recovery is something that also derives from your type system.

Re: Cloudflare outage on November 18, 2025 post mortem

#904

Post by frumplestlatz »

hedora wrote: post #306
evil-olive wrote: post #168

Earlier quoted context omitted.

> What happens to it up the callstack? as they say in the post, these files get generated every 5 minutes and rolled out across their fleet. so in this case, the thing farther up the callstack is a "watch for updated files and ingest them" component. that component, when it receives the error, can simply continue using the existing file it loaded 5 minutes earlier. and then it can increment a Prometheus metric (or si…

Given that the bug was elsewhere in the system (the config file parser spuriously failed), it’s hard to justify much of what you suggested. Panics should be logged, and probably grouped by stack trace for things like prometheus (outside of process). That handles all sorts of panic scenarios, including kernel bugs and hardware errors, which are common at cloudflare scale. Similarly, mitigating by having rapid restart…

Writing bad code that doesn’t handle errors and doesn’t correctly model your actual runtime invariants doesn’t simplify anything other than the amount of thought you have to put into writing the code — because you’re writing broken code.

The solution to this problem wasn’t restarting the failing process. It was correctly modeling the failure case, so that then the type system forced you to correctly handle it.

Re: Cloudflare outage on November 18, 2025 post mortem

#905

Post by Yokohiii »

michaelt wrote: post #622
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> Why did they think this was a cyberattack Isn’t getting cyberattacked their core business?

If so, why is their discovery of, non ambiguous?

Re: Cloudflare outage on November 18, 2025 post mortem

#906

Post by rixed »

frumplestlatz wrote: post #903
rixed wrote: post #463

Earlier quoted context omitted.

Exactly! You could have `rand() > 0.5 && panic!()` in the code of your bot module, and that should not put the internet on fire. The issue here is about the system as a whole not any line of code.

> The issue here is about the system as a whole not any line of code. Unsoundness in the type system that leads to a systemic failure is about the system as a whole. Not everything can be recovered from restarting a process, and process correctness and recovery is something that also derives from your type system.

In the early 2000s when Google explained how they achieved their (already back then) awesome reliability, ie assuming that any software and hardware will eventually fail, and that they designed everything with the idea that everything was faulty, there were some people who couldn't get it, who would still bring the argument that "yeah but today with modern raid..."

People here chatting about unwrap remind me of them :)

Re: Cloudflare outage on November 18, 2025 post mortem

#907

Post by Yokohiii »

aforwardslash wrote: post #874
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

I know its easy to criticize what happened after the fact and having a clear(er) picture of all the moving parts and the timeline of events, but I think that while most of the people in the thread are pointing out either Rust-related or lack of configuration validation, what really grinds my gears is something that - in my opinion - is bad engineering. Having an unprivileged application querying system.columns to inf…

On all the other issues, I thought they wanted to do the right thing at heart, but missed to make it fail safe. I can pass it as a problem of a journey to maturity or simply the fact that you can't get everything perfect. Maybe even a bit of sloppiness here and there.

The database issue screamed at me: lack of expertise. I don't use CH, but seeing someone to mess with a production system and they being surprised "Oh, it does that?", is really bad. And this is obviously not knowledge that is hard to achieve, buried deep in a manual or an edge case only discoverable by source code, it's bread and butter knowledge you should know.

What is confusing, that they didn't add this to their follow-up steps. With some benefit of doubt I'd assume they didn't want to put something very basic as a reason out there, just to protect the people behind it from widespread blame. But if that's not the case, then it's a general problem. Sadly it's not uncommon that components like databases are dealt with, on an low effort basis. Just a thing we plug in and works. But it's obviously not.

Re: Cloudflare outage on November 18, 2025 post mortem

#908

Post by Yokohiii »

nip wrote: post #748
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

It’s easy to pick on logic that failed and for which you have a very detailed and great post mortem write-up. Yet you omit to acknowledge that the remaining 99.99999% logic written that powers Cloudflare works flawlessly. Also, hindsight is 20/20

You are less critical with CF then they are with themselves.

A system that is 99.99999% flawless, can still be unusable.

optimism bias: 100/100

Re: Cloudflare outage on November 18, 2025 post mortem

#909

Post by mike_hearn »

bombela wrote: post #890
mike_hearn wrote: post #781

Earlier quoted context omitted.

It's one of the problems with using result types. You don't distinguish between genuinely exceptional events and things that are expected to happen often on hot paths, so the runtime doesn't know how much data to collect.

panic is the exceptional event. It so happens that rust doesn't print a stacktrace in release unless configured to do so. Similarly, capturing a stack trace in a error type (within a Result for example) is perfectly possible. But this is a choice left to the programmer, because capturing a trace is not cheap.

There's clearly a big gap in how things are done in practice. You wouldn't see anyone call System.exit in a managed language if a data file was bigger than expected. You'd always get an exception.

I used to be an SRE at Google. Back then we also had big outages caused by bad data files pushed to prod. It's a common enough issue so I really sympathize with Cloudflare, it's not nice to be on call for issues like that. But Google's prod environments always generated stack traces for every kind of failure, including CHECK failures (panics) in C++. You could also reflect the stack traces of every thread via HTTP. I used to diagnose bugs in production under time pressure quite regularly using just these tools. You always need detailed diagnostics.

Languages shouldn't have panics, tbh, it's a primitive concept. It so rarely makes sense to handle errors that way. I know there's a whole body of Rust/Go lore claiming panics are fine, but it's not a good move and is one of the reasons I've stayed away from Go over the years and wouldn't use Rust for anything higher than low level embedded components or operating system code that has to export a C ABI. You always want diagnostics and recoverable errors; this kind of micro-optimization doesn't make sense outside of extremely constrained embedded environments that very few of us work in.

Re: Cloudflare outage on November 18, 2025 post mortem

#910

Post by StopDisinfo910 »

JuniperMesos wrote: post #861
StopDisinfo910 wrote: post #596

Earlier quoted context omitted.

> I honestly have a hard time suggesting that those customers are individually wrong to do so - Cloudflare isn't down that often, and whatever amount it cost any individual customer by being down today might be more than offset by the DDOS protection they're buying. We have collectively agreed to a world where software service providers have no incentive to be reliable as they are shielded from the consequences of th…

> I want software provider to be liable for the damage they cause and minimum quality regulation on par with an actual engineering discipline. I have always been astounded that nearly all software licences start with extremely broad limitation of liability provisions and people somehow feel fine with it. Try to extend that to any other product you regularly use in your life and see how that makes you fell. So do you…

I absolutely do not mind, yes.

You can't go out in the middle of your city, build a shoddy bridge, say you wave all responsibilities and then wash your hands with the consequences when it predictably breaks. Why can you do that with pieces of software?

Limiting the scope of liability waivers is not the same things as censoring what software can be produced. It's just ensuring that everyone actually take responsibility for the things they distribute.

As I said previously, the current situation doesn't make sense to me. People have been brainwashed in believing that the way software is released currently, half finished and crippled with bugs, is somehow normal and acceptable. It absolutely doesn't have to be this way.

It'a beyond shameful that the average developers today is blissfully unaware of anything related to producing actually secure pieces of software. I am pretty sure I can walk into more than 90% of development shops today and no one there will know what formal methods are. With some luck, they might have some static analysers running, probably from a random provider and be happy with the crappy percentages that it outputs.

It's not about research. It's about a field which entirely refuses to become mature despite being pivotal to the modern economy. And why would it? Software products somehow get a free pass for the shit they push on everyone.

We are in the classical "market for lemons" trap where negative externalities are not priced in and investing in security will just get you to lose against companies that don't care. Every major incidents remind us we need out. The market has already showed it won't self correct. It's a classical case where regulatory intervention is necessary and legitimate.

The shift is already happening by the way. The EU product liability directive was adopted in 2024 and the transition period ends in December 2026. The US "National Cybersecurity Strategy" signals intend to review the status quo. It's coming faster that people realise.

Post reply on HN