Cloudflare outage on December 5, 2025 • Hacker News Board

Re: Cloudflare outage on December 5, 2025

#301

Post by miyuru » Fri, Dec 05, 2025, 9:16 PM UTC

LelouBil wrote: post #168

miyuru wrote: post #16

Whats going on with cloudflare's software team? I have seen similar bugs in cloudflare API recently as well. There is an endpoint for a feature that is available only to enterprise users, but the check for whether the user is on an enterprise plan is done at the last step.

Can you elaborate? I'm not sure what you mean by "at the last step"

The API endpoint I am talking about needs a external verification. they allow to do the external verification before checking if the user is on the enterprise plan or not.

The feature is only available to enterprise plans, it should not even allow external verification.

Re: Cloudflare outage on December 5, 2025

#302

Post by venturecruelty » Fri, Dec 05, 2025, 9:16 PM UTC

8cvor6j844qw_d6 wrote: post #102

Is there some underlying factors that resulted in the recent outages (e.g., new processes, layoffs, etc.) or just a series of pure coincidences?

I'm sure everything slowly falling apart all at the same time is due to some strange coincidence, and not the regular and steady firing of thousands of people.

Re: Cloudflare outage on December 5, 2025

#303

Post by djmips » Fri, Dec 05, 2025, 9:16 PM UTC

vouwfietsman wrote: post #253

uyzstvqs wrote: post #122

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately roll…

> Languages with strong type systems won't save you Neither will seatbelts if you drive into the ocean, or helmets if you drink poison. I'm not sure what your point is.

I think you strengthened their point.

Re: Cloudflare outage on December 5, 2025

#304

Post by venturecruelty » Fri, Dec 05, 2025, 9:17 PM UTC

gkoz wrote: post #26

I sometimes feel we'd be better off without all the paternalistic kitchensink features. The solid, properly engineered features used intentionally aren't causing these outages.

The good news is that you can have that right now. Just don't use Cloudflare.

Re: Cloudflare outage on December 5, 2025

#305

Post by ljm » Fri, Dec 05, 2025, 9:18 PM UTC

morpheos137 wrote: post #268

testplzignore wrote: post #179

Earlier quoted context omitted.

> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions. Their timeline: > 08:47: Configuration change deplo…

I see lots of people complaining about this down time but in actuality is it really that big a deal to have 30 minutes of down time or whatever. It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake. In many developed countries the electric power service has local down times on occasion. That's more important than not being able…

> It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake.

This is far too dismissive of how disruptive the downtime can be and it sets the bar way too low for a company so deeply entangled in global internet infrastructure.

I don’t think you can make such an assertion with any degree of credibility.

Re: Cloudflare outage on December 5, 2025

#306

Post by venturecruelty » Fri, Dec 05, 2025, 9:18 PM UTC

dznodes wrote: post #215

When should we just give up on Cloudflare? Seems like this just keeps happening. Like some kind of backdoor triggered willy nilly, Hmmm?

Now. Right now. Seriously, stop using this terrible service. We also need to change the narrative that step 1 in every tutorial is "sign up for Cloudflare". This is partly a culture problem.

Re: Cloudflare outage on December 5, 2025

#307

Post by odie5533 » Fri, Dec 05, 2025, 9:19 PM UTC

morpheos137 wrote: post #268

testplzignore wrote: post #179

Earlier quoted context omitted.

> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions. Their timeline: > 08:47: Configuration change deplo…

I see lots of people complaining about this down time but in actuality is it really that big a deal to have 30 minutes of down time or whatever. It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake. In many developed countries the electric power service has local down times on occasion. That's more important than not being able…

> It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake.

Yes, there are lots of mission critical systems that use cloudflare and lives and huge amounts of money are at stake.

Re: Cloudflare outage on December 5, 2025

#308

Post by shadowgovt » Fri, Dec 05, 2025, 9:22 PM UTC

Scaevolus wrote: post #6

> Disabling this was done using our global configuration system. This system does not use gradual rollouts but rather propagates changes within seconds to the entire network and is under review following the outage we recently experienced on November 18. > As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following LUA exception: They r…

"Hey, this change is making the 'check engine' light turn on all the time. No problem; I just grabbed some pliers and crushed the bulb."

Re: Cloudflare outage on December 5, 2025

#309

Post by odie5533 » Fri, Dec 05, 2025, 9:22 PM UTC

lionkor wrote: post #101

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

When a piece of hardware goes or a careless backup process fails, downtime of a self-hosted service can be measured in days or weeks.

Re: Cloudflare outage on December 5, 2025

#310

Post by aw1621107 » Fri, Dec 05, 2025, 9:24 PM UTC

da_grift_shift wrote: post #27

It's not an outage, it's an Availability Incident™. https://blog.cloudflare.com/5-december-2025-outage/#what-abo...

From earlier in the very same blog post (emphasis added):

> This system does not perform gradual rollouts, but rather propagates changes within seconds to the entire fleet of servers in our network and is under review following the outage we experienced on November 18.