Cloudflare outage on December 5, 2025 • Hacker News Board

Re: Cloudflare outage on December 5, 2025

#281

Post by dkyc » Fri, Dec 05, 2025, 8:33 PM UTC

flaminHotSpeedo wrote: post #100

dkyc wrote: post #75

Earlier quoted context omitted.

One thing to keep in mind when judging what's 'appropriate' is that Cloudflare was effectively responding to an ongoing security incident outside of their control (the React Server RCE vulnerability). Part of Cloudlfare's value proposition is being quick to react to such threats. That changes the equation a bit: any hour you wait longer to deploy, your customers are actively getting hacked through a known high-severi…

To clarify, I'm not trying to imply that I definitely wouldn't have made the same decision, or that cowboy decisions aren't ever the right call. However, this preliminary report doesn't really justify the decision to use the same deployment system responsible for the 11/18 outage. Deployment safety should have been the focus of this report, not the technical details. My question that I want answered isn't "are there…

The 11/18 outage was 2.5 weeks ago. Any learning & changes they made as a result for that probably didn't make its way yet to production.

Particularly if we're asking them to be careful & deliberate about deployments, hard to ask them fast-track this.

Re: Cloudflare outage on December 5, 2025

#282

Post by hinkley » Fri, Dec 05, 2025, 8:36 PM UTC

vpShane wrote: post #248

zwnow wrote: post #197

Earlier quoted context omitted.

"Kudos"? This is like the South Park episode in which the oil company guy just excuses himself while the company just continues to fuck up over and over again. There's nothing to praise, this shouldn't happen twice in a month. Its inexcusable.

twice in a month _so far_

We still have two holidays and associated vacations and vacation brain to go. And then the January hangover.

Every company that has ignored my following advice has experienced a day for day slip in first quarter scheduling. And that advice is: not much work gets done between Dec 15 and Jan 15. You can rely on a week worth, more than that is optimistic. People are taking it easy and they need to verify things with someone who is on vacation so they are blocked. And when that person gets back, it’s two days until their vacation so it’s a crap shoot.

NB: there’s work happening on Jan 10, for certain, but it’s not getting finished until the 15th. People are often still cleaning up after bad decisions they made during the holidays and the subsequent hangover.

Re: Cloudflare outage on December 5, 2025

#283

Post by DANmode » Fri, Dec 05, 2025, 8:36 PM UTC

markus_zhang wrote: post #177

lionkor wrote: post #101

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

TBF, it depends on the number of outages locally. In my area it is one outage every thunderstorm/snow storm, so unfortunately the up time of my laptop, even with the help of a large, portable battery charging station (which can charge multiple laptops at the same time), is not optimistic. I sometimes fancy that I could just take cash, go into the wood, build a small solar array, collect & cleanse river water, and buy…

Far more achievable pricing and logistics than even ten years ago.

Re: Cloudflare outage on December 5, 2025

#284

Post by switchbak » Fri, Dec 05, 2025, 8:36 PM UTC

lloeki wrote: post #260

SketchySeaBeast wrote: post #252

Earlier quoted context omitted.

I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.

By reading the documentation thoroughly as a compulsory first step to designing the next system that depends on it. I realise this may probably boggle the mind of the modern software developer.

Just try harder. And if it still breaks, clearly you weren't trying hard enough!

At some point you have to admit that humans are pretty bad at some things. Keeping documentation up to date and coherent is one of those things, especially in the age of TikTok.

Better to live in the world we have and do the best you can, than to endlessly argue about how things should be but never will become.

Re: Cloudflare outage on December 5, 2025

#285

Post by bombcar » Fri, Dec 05, 2025, 8:40 PM UTC

Scaevolus wrote: post #6

> Disabling this was done using our global configuration system. This system does not use gradual rollouts but rather propagates changes within seconds to the entire network and is under review following the outage we recently experienced on November 18. > As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following LUA exception: They r…

“ Uh...it's probably not a problem...probably...but I'm showing a small discrepancy in...well, no, it's well within acceptable bounds again. Sustaining sequence. Nothing you need to worry about, Gordon. Go ahead.“

Re: Cloudflare outage on December 5, 2025

#286

Post by bombcar » Fri, Dec 05, 2025, 8:42 PM UTC

morpheos137 wrote: post #268

testplzignore wrote: post #179

Earlier quoted context omitted.

> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions. Their timeline: > 08:47: Configuration change deplo…

I see lots of people complaining about this down time but in actuality is it really that big a deal to have 30 minutes of down time or whatever. It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake. In many developed countries the electric power service has local down times on occasion. That's more important than not being able…

30 minutes of downtime is fine for most things, including Amazon.

30 minutes of unplanned downtime for infrastructure is unacceptable; but we’re tending to accept it. AWS or Cloudflare have positioned themselves as The Internet so they need to be held to a higher standard.

Re: Cloudflare outage on December 5, 2025

#287

Post by bombcar » Fri, Dec 05, 2025, 8:44 PM UTC

XCSme wrote: post #254

ryandvm wrote: post #247

Earlier quoted context omitted.

Indeed. AWS too. I feel like the cloud hosting companies have lost the plot. "They can provide better uptime than us" is the entire rationale that a lot of small companies have when choosing to run everything in the cloud. If they cost more AND they're less reliable, what exactly is the reason to not self host?

Plus, when you self-host, you can likely fix the issue yourself in a couple of hours max, instead of waiting indefinitely for a fix or support that might never come.

These global cloud outages aren’t the real issue; they affect everyone and get fixed.

What is killer is when there is a KNOWN issue that affects YOU but basically only you so why bother fixing it!

Re: Cloudflare outage on December 5, 2025

#288

Post by jacobgkau » Fri, Dec 05, 2025, 8:45 PM UTC

I noticed this outage last night (Cloudflare 500s on a few unrelated websites). As usual, when I went to Cloudflare's status page, nothing about the outage was present; the only thing there was a notice about the pre-planned maintenance work they were doing for the security issue, reporting that everything was being routed around it successfully.

Re: Cloudflare outage on December 5, 2025

#289

Post by bombcar » Fri, Dec 05, 2025, 8:45 PM UTC

uyzstvqs wrote: post #122

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately roll…

They have millions of “free” subscribers; said subscribers should be the test pigs for rollouts; paying (read: big) subscribers can get the breaking changes later.

Re: Cloudflare outage on December 5, 2025

#290

Post by wizzwizz4 » Fri, Dec 05, 2025, 8:48 PM UTC

deadbabe wrote: post #145

udev4096 wrote: post #110

Earlier quoted context omitted.

[dead]

It is healthy for tech companies to have outages, as they will build experience in resolving them. Success breeds complacency.

You don't need outages to build experience in resolving them, if you identify conditions that increase the risk of outages. Airlines can develop a lot of experience resolving issues that would lead to plane crashes, without actually crashing any planes.