Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

291–300 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#291

Post by hedora »

otterley wrote: post #269
HumanOstrich wrote: post #246

Earlier quoted context omitted.

You didn't really put any thought into what I said. Thanks for the links.

It wasn't worth thinking about. I'm not going to defend myself against arguments and absolute claims I didn't make. The key word here is mitigation, not perfection.

> If your AWS service is properly regionalized, that’s the minimum amount of cellular architecture required

Amazon has had multi-region outages due to pushing bad configs, so it’s extremely difficult to believe whatever you are proposing solves that exact problem by relying on multi-regions.

Come to think of it, Cloudflare’s outage today is another good counterexample.

Re: Cloudflare outage on November 18, 2025 post mortem

#292

Post by chrisweekly »

jcalvinowens wrote: post #275
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

semantic? or pedantic?

Re: Cloudflare outage on November 18, 2025 post mortem

#293

Post by yoyohello13 »

nomel wrote: post #217
yoyohello13 wrote: post #206

Earlier quoted context omitted.

There are many many pages of text discussing this topic, but having programmed in both styles, exceptions make it too easy for programmer to simply ignore them. Errors as values force you to explicitly handle it there, or toss it up the stack. Maybe some other languages have better exception handling but in Python it’s god awful. In big projects you can basically never know when or how something can fail.

I would claim the opposite. If you don't catch an exception, you'll get a halt. With return values, you can trivially ignore an exception. let _ = fs::remove_file("file_doesn't_exist"); or value, error = some_function() // carry on without doing anything with error In the wild, I've seen far more ignoring return errors, because of the mechanical burden of having type handling at every function call. This is backed by…

I can agree to disagree :)

Re: Cloudflare outage on November 18, 2025 post mortem

#294

Post by prawn »

eastdakota wrote: post #35
gucci-on-fleek wrote: post #13

> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was tri…

Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.

Just asking out of curiosity, but roughly how many staff would've been involved in some way in sorting out the issue? Either outside regular hours or redirected from their planned work?

Re: Cloudflare outage on November 18, 2025 post mortem

#295

Post by abigailphoebe »

echelon wrote: post #288
jcalvinowens wrote: post #275

Earlier quoted context omitted.

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

[flagged]

blaming the language is not the way to approach this. if an engineer writes bad code that’s the engineers fault, not the languages.

this was bad code that should have never hit production, it is not a rust language issue.

Re: Cloudflare outage on November 18, 2025 post mortem

#296

Post by philipwhiuk »

bayesnet wrote: post #11
SerCe wrote: post #9

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

And a well-written one at that. Compared to the AWS port-mortem this could be literature.

Except it fails to document anything about the actions they made to Warp in London during the resolution.

Re: Cloudflare outage on November 18, 2025 post mortem

#297

Post by echelon »

abigailphoebe wrote: post #295
echelon wrote: post #288

Earlier quoted context omitted.

[flagged]

blaming the language is not the way to approach this. if an engineer writes bad code that’s the engineers fault, not the languages. this was bad code that should have never hit production, it is not a rust language issue.

No. Don't say "you're holding it wrong". The language says "safe" on the tin. It advertises safety. This shouldn't be possible.

This is a null pointer. In Rust.

Unwrap needs to die. We should all fight to remove it.

Re: Cloudflare outage on November 18, 2025 post mortem

#298

Post by spprashant »

bri3d wrote: post #260

Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random…

I have nowhere near the experience managing such complex systems, but I can empathize with this. In a high-pressure situations the most obvious things get missed. If someone is convinced System X is at fault, your mind can make leaps to justify every other degraded system is a downstream effect of that. Cause and effect can get switched.

Sometimes you have smart people in the room who dig deeper and fish it out, but you cannot always rely on that.

Re: Cloudflare outage on November 18, 2025 post mortem

#299

Post by HumanOstrich »

hedora wrote: post #286
HumanOstrich wrote: post #113

Earlier quoted context omitted.

I don't think sqlite would come close to their requirements for permissions or resilience, to name a couple. It's not the solution for every database issue. Also, the link you provided is for eventual deduplication at the storage layer, not deduplication at query time.

I think the idea is to ship the sqlite database around. It’s not a terrible idea, in that you can test the exact database engine binary in CI, and it’s (by definition) not a single point of failure.

I think you're oversimplifying the problem they had, and I would encourage you to dive in to the details in the article. There wasn't a problem with the database, it was with the query used to generate the configs. So if an analogous issue arose with a query against one of many ad-hoc replicated sqlite databases, you'd still have the failure.

I love sqlite for some things, but it's not The One True Database Solution.

Re: Cloudflare outage on November 18, 2025 post mortem

#300

Post by navidmafi »

jcalvinowens wrote: post #275
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

And dare I say, an exhibition of hindsight bias.
Post reply on HN