Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

241–250 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#241

Post by jbmsf »

eastdakota wrote: post #32
lukan wrote: post #20

"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our stat…

We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.

It looks a lot like a CloudFront error we randomly saw today from one of our engineers in South America. I suspect there was a small outage in AWS but can't prove it.

Re: Cloudflare outage on November 18, 2025 post mortem

#242

Post by duped »

pdimitar wrote: post #224
duped wrote: post #218

Earlier quoted context omitted.

This is not a reasonable take to me. unwrap/expect are the idiomatic way to express code paths returning Option/Result as unreachable. Bubbling up the error or None does not make the program correct. Panicking may be the only reasonable thing to do. If panicking is guaranteed because of some input mistake to the system your failure is in testing.

I agree the failure is in testing but what you can and should do is raise in alert in your APM system before the runtime panic, in the code path that is deemed impossible to hit. I am not trashing on them, I've made such mistakes in the past, but I do expect more from them is all. And you will not believe how many alerts I got for the "impossible" errors. I do agree there was not too much that could have been done, y…

There's certainly a discipline involved here, but it's usually something like guaranteeing all threads are unwind safe (via AssertUnwindSafe) and logging stack traces when your process keeps dying/can't be started after a fixed number of retries. Which would lead you to the culprit immediately.

I'm just pushing back a bit on the idea that unwrap() is unsafe - it's not, and I wouldn't even call it a foot gun. The code did what it was written to do, when it saw the input was garbage it crashed because it couldn't make sense of what to do next. That's a desirable property in reliable systems (of course monitoring that and testing it is what makes it reliable/fixable in the first place).

Re: Cloudflare outage on November 18, 2025 post mortem

#243

Post by [deleted] »

gpm wrote: post #181
cvhc wrote: post #122

Earlier quoted context omitted.

Some languages and style guides simply forbid throwing exceptions without catching / proper recovery. Google C++ bans exceptions and the main mechanism for propogating errors is `absl::Status` which the caller has to check. Not familiar with Rust but it seems unwrap is such a thing that would be banned.

Unwrap is used in places where in C++ you would just have undefined behavior. It wouldn't make any more sense to blanket ban it than it would ban ever dereferencing a pointer just in case its null - even if you just checked that it wasn't null.

[deleted]

Re: Cloudflare outage on November 18, 2025 post mortem

#244

Post by nwellinghoff »

The real take away is that so much functionality depends on a few players. This is a fundamental flaw in design that is getting worse by the year as the winner takes all winners win. Not saying they didn’t earn their wins. But the fact remains. The system is not robust. Then again, so what. It went down for a while. Maybe we shouldn’t depend on the internet being “up” all the time.

Re: Cloudflare outage on November 18, 2025 post mortem

#245

Post by TehShrike »

habibur wrote: post #127

On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures As of 17:06 all systems at Cloudflare were functioning as normal 6 hours / 5 years gives ~99.98% uptime.

I'm feeling generous tonight, I'm willing to consider 0.99986 to round to 99.99%

Re: Cloudflare outage on November 18, 2025 post mortem

#246

Post by HumanOstrich »

otterley wrote: post #235
HumanOstrich wrote: post #226

Earlier quoted context omitted.

What variant of cellular architecture are you referring to? Can you give me a link or few? I'm fascinated by it and I've led a team to break up a monolithic solution running on AWS to a cellular architecture. The results were good, but not magic. The process of learning from failures did not stop, but it did change (for the better). No matter what architecture, processes, software, frameworks, and systems you use, or…

If your AWS service is properly regionalized, that’s the minimum amount of cellular architecture required. Did your service ever fail in multiple regions simultaneously? Cellular architecture within a region is the next level and is more difficult, but is achievable if you adhere to the same principles that prohibit inter-regional coupling: https://docs.aws.amazon.com/wellarchitected/latest/reducing-... https://docs.…

You didn't really put any thought into what I said. Thanks for the links.

Re: Cloudflare outage on November 18, 2025 post mortem

#247

Post by vlovich123 »

hedora wrote: post #186
vlovich123 wrote: post #162

Earlier quoted context omitted.

To be fair, this failed in the non-rust path too because the bot management returned that all traffic was a bot. But yes, FL2 needs to catch panics from individual components but I’m not sure if failing open is necessarily that much better (it was in this case but the next incident could easily be the result of failing open). But more generally you could catch the panic at the FL2 layer to make that decision intentio…

Catching panic probably isn’t a great idea if there’s any unsafe code in the system. (Do the unsafe blocks really maintain heap invariants if across panics?)

Unsafe blocks have nothing to do with it. Yes - they maintain all the same invariants as safe blocks or those unsafe blocks are unsound regardless of panics. But there’s millions of way to architect this (eg a supervisor process that notices which layer in FL2 is crashing and just completely disables that layer when it starts up the proxy again. There’s challenges here because then you have to figure out what constitutes a perma crashing (eg what if it’s just 20% of all sites? Do you disable?). And in the general case you have the fail open/fail close decision anyway which you should just annotate individual layers with.

But the bigger change is to make sure that config changes roll out gradually instead of all at once. That’s the source of 99% of all widespread outages

Re: Cloudflare outage on November 18, 2025 post mortem

#248

Post by pdimitar »

duped wrote: post #242
pdimitar wrote: post #224

Earlier quoted context omitted.

I agree the failure is in testing but what you can and should do is raise in alert in your APM system before the runtime panic, in the code path that is deemed impossible to hit. I am not trashing on them, I've made such mistakes in the past, but I do expect more from them is all. And you will not believe how many alerts I got for the "impossible" errors. I do agree there was not too much that could have been done, y…

There's certainly a discipline involved here, but it's usually something like guaranteeing all threads are unwind safe (via AssertUnwindSafe) and logging stack traces when your process keeps dying/can't be started after a fixed number of retries. Which would lead you to the culprit immediately. I'm just pushing back a bit on the idea that unwrap() is unsafe - it's not, and I wouldn't even call it a foot gun. The code…

We don't disagree, my main point was a bit broader and admittedly hijacked the original topic a bit, namely: `unwrap` and `expect` make many Rust devs too comfortable and these two are very tempting mistresses.

Using those should be done in an extremely disciplined manner. I agree that there are many legitimate uses but in the production Rust code I've seen this has rarely been the case. People just want to move on and then forget to circle back and add proper error handling. But yes, in this case that's not quite true. Still, my point that an APM alert should have been raised on the "impossible" code path before panicking, stands.

Re: Cloudflare outage on November 18, 2025 post mortem

#249

Post by nrhrjrjrjtntbt »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I wonder what happens if they handle it gracefully? sounds like performance degradation (better than reliability degradation!).

Also wonder with a sharded system why are they not slow rolling out changes and monitoring?

Re: Cloudflare outage on November 18, 2025 post mortem

#250

Post by crote »

ajross wrote: post #183
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I'm not completely sure I agree. I mean, I do agree about the .unwrap() culture being a bug trap. But I don't think this example qualifies. The root cause here was that a file was mildly corrupt (with duplicate entries, I guess). And there was a validation check elsewhere that said "THIS FILE IS TOO BIG". But if that's a validation failure, well, failing is correct? What wasn't correct was that the failure reached pr…

It sounds to me like there was validation, but the system wasn't designed for the validation to ever fail - at which point crashing is the only remaining option. You've essentially turned it into an assertion error rather than a parsing/validation error.

Ideally every validation should have a well-defined failure path. In the case of a config file rotation, validation failure of the new config could mean keeping the old config and logging a high-priority error message. In the case of malformed user-provided data, it might mean dropping the request and maybe logging it for security analysis reasons. In the case of "pi suddenly equals 4" checks the most logical approach might be to intentionally crash, as there's obviously something seriously wrong and application state has corrupted in such a way that any attempt to continue is only going to make things worse.

But in all cases there's a reason behind the post-validation-failure behavior. At a certain point leaving it up to "whatever happens on .unwrap() failure" isn't good enough anymore.

Post reply on HN