Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

361–370 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#361

Post by [deleted] »

jcalvinowens wrote: post #317
AdieuToLogic wrote: post #309

Earlier quoted context omitted.

>> This is the multi-million dollar .unwrap() story. > That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same. Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assu…

> Instead, the routine likely should have worked within `Result`. But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same. Maybe you're saying "it's better to be explicit", as a broad generalization I don't disagree with that. But that has nothing to do with the actual bug here, which was that the invariant failed. How they choose to implement checking and failing the invar…

[deleted]

Re: Cloudflare outage on November 18, 2025 post mortem

#362

Post by rafaelmn »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

That's such a bad take after reading the article. If you're going to write a system that preallocates and is based on hard assumptions about max size - the panic/unwrap approach is reasonable.

The config bug reaching prod without this being caught and pinpointed immediately is the strange part.

Re: Cloudflare outage on November 18, 2025 post mortem

#363

Post by gizmo686 »

forsalebypwner wrote: post #150
dbetteridge wrote: post #124

Earlier quoted context omitted.

Question from a casual bystander, why not have a virtual/staging mini node that receives these feature file changes first and catches errors to veto full production push? Or you do have something like this but the specific db permission change in this context only failed in production

I think the reasoning behind this is because of the nature of the file being pushed - from the post mortem: "This feature file is refreshed every few minutes and published to our entire network and allows us to react to variations in traffic flows across the Internet. It allows us to react to new types of bots and new bot attacks. So it’s critical that it is rolled out frequently and rapidly as bad actors change thei…

In this case, the file fails quickly. A pretest that consists of just attempting to load the file would have caught it. Minutes is more than enough time to perform such a check.

Re: Cloudflare outage on November 18, 2025 post mortem

#364

Post by kevin_thibedeau »

rafaelmn wrote: post #362
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

That's such a bad take after reading the article. If you're going to write a system that preallocates and is based on hard assumptions about max size - the panic/unwrap approach is reasonable. The config bug reaching prod without this being caught and pinpointed immediately is the strange part.

It's reasonable when testing protocols exercise the panic scenario. This is the problem with punting on error recovery. Nobody checks faults that propagate across domains of responsibility.

Re: Cloudflare outage on November 18, 2025 post mortem

#365

Post by tonyhart7 »

aw1621107 wrote: post #337
tonyhart7 wrote: post #201

Earlier quoted context omitted.

and people make mistake at some point machine would be better in coding because well machine code is machine instruction task same like chess, engine is better than human grandmaster because its solvable math field coding is no different

> same like chess, engine is better than human grandmaster because its solvable math field Might be worth noting that your description of chess is slightly incorrect. Chess technically isn't solved in the sense that the optimal move is known for any arbitrary position is known; it's just that chess engines are using what amounts to a fancy brute force for most of the game and the combination of hardware and search al…

No ?????? because these thing called BEST MOVE and BAD MOVE there in chess

"chess engines are still capable of making mistakes", I'm sorry no

inaccurate yes but not mistake

Re: Cloudflare outage on November 18, 2025 post mortem

#366

Post by Pulcinella »

monocularvision wrote: post #358
ChrisMarshallNY wrote: post #190

Earlier quoted context omitted.

Swift has implicit unwrap (!), and explicit unwrap (?). I don't like to use implicit unwrap. Even things that are guaranteed to be there, I treat as explicit (For example, (self.view?.isEnabled ?? false) , in a view controller, instead of self.view.isEnabled ). I always redefine @IBOutlets from: @IBOutlet weak var someView! to: @IBOutlet weak var someView? I'm kind of a "belt & suspenders" type of guy.

So what happens if it ends up being nil? How does your app react? In this particular case, I would rather crash. It’s easier to spot in a crash report and you get a nice stack trace. Silent failure is ultimately terrible for users. Note: for the things I control I try to very explicitly model state in such a way as I never need to force unwrap at all. But for things beyond my control like this situation, I would rath…

Yeah @IBOutlets are generally the one thing that are allowed to be implicitly-unwrapped optionals. They go along with using storyboards & xibs files with Interface Builder. I agree that you really should just crash if you are attempting to access one and it is nil. Either you have done something completely incorrect with regards to initializing and accessing parts of your UI and want to catch that in development, or something has gone horribly, horribly, horribly with UIKit/AppKit and storyboard/xib files are not being loaded properly by the system.

Re: Cloudflare outage on November 18, 2025 post mortem

#367

Post by aetherspawn »

Cloudflare Access is still experiencing weird issues for us (it’s asking users to SSO login to our public website even though our zone rules - set on a completely different zone - haven’t changed).

I don’t think the infrastructure has been as fully recovered as they think yet…

Re: Cloudflare outage on November 18, 2025 post mortem

#368

Post by 0xbadcafebee »

tptacek wrote: post #24
0xbadcafebee wrote: post #4

So, to recap: - Their database permissions changed unexpectedly (??) - This caused a 'feature file' to be changed in an unusual way (?!) - Their SQL query made assumptions about the database; their permissions change thus resulted in queries getting additional results, permitted by the query - Changes were propagated to production servers which then crashed those servers (meaning they weren't tested correctly) - They…

People jump to say things like "where's the rollback" and, like, probably yeah, but keep in mind that speculative rollback features (that is: rollbacks built before you've experienced the real error modes of the system) are themselves sources of sometimes-metastable distributed system failures. None of this is easy.

How about where's the most basic test to check if your config file will actually run at all in your application? It was a hard-coded memory limit; a git-hook test suite run a MacBook would have caught this. But nooo, let's not run the app for 0.01 seconds with this config before sending it out to determine the fate of the internet?

This is literally the CrowdStrike bug, in a CDN. This is the most basic, elementary, day 0 test you could possibly invent. Forget the other things they fucked up. Their app just crashes with a config file, and nobody evaluates it?! Not every bug is preventable, but an egregious lack of testing is preventable.

This is what a software building code (like the electrical code's UL listings that prevent your house from burning down from untested electrical components) is intended to prevent. No critical infrastructure should be legal without testing, period.

Re: Cloudflare outage on November 18, 2025 post mortem

#369

Post by anitil »

richardlblair wrote: post #321
eastdakota wrote: post #95

Earlier quoted context omitted.

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…

A very human and authentic response. Love to see it. Fantastic for recruiting, too.

> He didn’t reply after a few minutes so I did

I'd consider applying based on this alone

Post reply on HN