Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

121–130 of 601 posts

Re: Cloudflare outage on December 5, 2025

#121

Post by 8cvor6j844qw_d6 »

jakub_g wrote: post #117

The interesting part: After rolling out a bad ruleset update, they tried a killswitch (rolled out immediately to 100%) which was a code path never executed before: > However, we have never before applied a killswitch to a rule with an action of “execute”. When the killswitch was applied, the code correctly skipped the evaluation of the execute action, and didn’t evaluate the sub-ruleset pointed to by it. However, an…

> have never before applied a killswitch to a rule with an action of “execute”

One might think a company on the scale of Cloudflare would have a suite of comprehensive tests to cover various scenarios.

Re: Cloudflare outage on December 5, 2025

#122

Post by uyzstvqs »

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario.

Only after that do you use gradual deployment, with a big red oopsie button which immediately rolls the changes back. Languages with strong type systems won't save you, good procedure will.

Re: Cloudflare outage on December 5, 2025

#124

Post by esseph »

RoyTyrell wrote: post #7
throwawaymaths wrote: post #4

Earlier quoted context omitted.

this time in lua. cloudflare can't catch a break

Or they're not thoroughly testing changes before pushing them out. As I've seen some others say, CloudFlare at this point should be considered critical infrastructure. Maybe not like power but dang close.

My power goes out every Wednesday around noon and normally if the weather is bad. In a major US metro.

I hope cloudflare is far more resilient than local power.

Re: Cloudflare outage on December 5, 2025

#125

Post by Bender »

Suggestion for Cloudflare: Create an early adopter option for free accounts.

Benefit: Earliest uptake of new features and security patches.

Drawback: Higher risk of outages.

I think this should be possible since they already differentiate between free, pro and enterprise accounts. I do not know how the routing for that works but I bet they could do this. Think crowd-sourced beta testers. Also a perk for anything PCI audit or FEDRAMP security prioritized over uptime.

Re: Cloudflare outage on December 5, 2025

#126

Post by nish__ »

ilkkao wrote: post #42
gkoz wrote: post #26

I sometimes feel we'd be better off without all the paternalistic kitchensink features. The solid, properly engineered features used intentionally aren't causing these outages.

Agreed, I don't really like Cloudflare trying to magically fix every web exploit there is in frameworks my site has never used.

Honestly. This feels outside of their domain.

Re: Cloudflare outage on December 5, 2025

#127

Post by nish__ »

denysvitali wrote: post #47
trashburger wrote: post #39

Earlier quoted context omitted.

I would very much like for him not to ignore the negativity, given that, you know, they are breaking the entire fucking Internet every time something like this happens.

This is the kind of comment I wish he would ignore. You can be angry - but that doesn't help anyone. They fucked up, yes, they admitted it and they provided plans on how to address that. I don't think they do these things on purpose. Of course given their good market penetration they end up disrupting a lot of customers - and they should focus on slow rollouts - but I also believe that in a DDOS protection system (or…

Maybe not on purpose but there's such a thing as negligence.

Re: Cloudflare outage on December 5, 2025

#128

Post by toomuchtodo »

udev4096 wrote: post #107
dkyc wrote: post #75

Earlier quoted context omitted.

One thing to keep in mind when judging what's 'appropriate' is that Cloudflare was effectively responding to an ongoing security incident outside of their control (the React Server RCE vulnerability). Part of Cloudlfare's value proposition is being quick to react to such threats. That changes the equation a bit: any hour you wait longer to deploy, your customers are actively getting hacked through a known high-severi…

[flagged]

Indeed, but it is what it is. Cloudflare comes out of my budget, and even with downtime, its better than not paying them. Do I want to deal with what Cloudflare offers? I do not, I have higher value work to focus on. I want to pay someone else to deal with this, and just like when cloud providers are down, it'll be back up eventually. Grab a coffee or beer and hang; we aren't savings lives, we're just building websites. This is not laziness or nihilism, but simply being rational and pragmatic.

Re: Cloudflare outage on December 5, 2025

#129

Post by perching_aix »

da_grift_shift wrote: post #27

It's not an outage, it's an Availability Incident™. https://blog.cloudflare.com/5-december-2025-outage/#what-abo...

You jest, but recently I also felt compelled to stop using the word (planned) outage where I work, because it legitimately creates confusion around the (expected) character of impact.

Outage is the nuclear wasteland situation, which given modern architectural choices, is rather challenging to manifest. To avoid it is face-saving, but also more correct.

Post reply on HN