Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

311–320 of 601 posts

Re: Cloudflare outage on December 5, 2025

#311

Post by al_borland »

XCSme wrote: post #295
bombcar wrote: post #287

Earlier quoted context omitted.

These global cloud outages aren’t the real issue; they affect everyone and get fixed. What is killer is when there is a KNOWN issue that affects YOU but basically only you so why bother fixing it!

I mean, I still prefer to have the ability to fix it myself, because I know I can probably do it in 1h max. I know this doesn't apply to most people, especially those outside of HN though.

Even if resolution times are equal, there is some comfort in being able to see the problem and make progress on it to feel like you're actively doing something. I work in a large enterprise and we have a team dedicated to managing critical incidents and getting everyone together for a resolution. When a 3rd party vendor is the reason for the outage, those calls are really awkward. It's a bunch of people sitting around pressing F5, all frantically trying to make it look like they are actively helping, when no one is actually doing anything, because they can't.

I equate it to driving. I'd rather be moving at a normal speed on side streets than sitting in traffic on the expressway, even if the expressway is technically faster.

Re: Cloudflare outage on December 5, 2025

#312

Post by PKop »

agentifysh wrote: post #232
liampulles wrote: post #90

The lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase. The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.

yet my dedicated server has been up since 2015 with zero downtimes i dont think this is an entropy issue its human error bubbling up and cloudflare charges a premium for it my faith in cloudflare is shoook for sure two major outages weeks apart ad this wont be the last

Why is the stability of your dedicated server a counterpoint that cloud behemoths can't keep up with their increasing entropy? Seems more like a supporting argument of OP at best, a non sequitur at worst.

Re: Cloudflare outage on December 5, 2025

#313

Post by SkyPuncher »

lloeki wrote: post #260
SketchySeaBeast wrote: post #252

Earlier quoted context omitted.

I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.

By reading the documentation thoroughly as a compulsory first step to designing the next system that depends on it. I realise this may probably boggle the mind of the modern software developer.

I used to take this approach when building new integrations. Then I realized (1) most documentation sucks (2) there's far too much to remember (3) much of it is conditional (4) you don't always know what matters until it matters (e.g. using different paths of implementation).

What works much better is having an intentional review step that you come back to.

Re: Cloudflare outage on December 5, 2025

#314

Post by ectospheno »

bombcar wrote: post #289
uyzstvqs wrote: post #122

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately roll…

They have millions of “free” subscribers; said subscribers should be the test pigs for rollouts; paying (read: big) subscribers can get the breaking changes later.

Free tier doesn’t get WAF. We kept working.

Re: Cloudflare outage on December 5, 2025

#315

Post by cnnlives265 »

jacobgkau wrote: post #288

I noticed this outage last night (Cloudflare 500s on a few unrelated websites). As usual, when I went to Cloudflare's status page, nothing about the outage was present; the only thing there was a notice about the pre-planned maintenance work they were doing for the security issue, reporting that everything was being routed around it successfully.

This is the case with just about every status page I’ve ever seen. It takes them a while to realize there’s really a problem and then to update the page. One day these things will be automated, but until then, I wouldn’t expect more of Cloudflare than any other provider.

What’s more concerning to me is that now we’ve had AWS, Azure, and CloudFlare (and CliudFlare twice) go down recently. My gut says:

1. developers and IT are using LLMs in some part of the process, which will not be 100% reliable.

2. Current culture of I have (some personal activity or problem) or we don’t have staff, AI will replace me, f-this.

3. Pandemic after effects.

4. Political climate / war / drugs; all are intermingled.

Re: Cloudflare outage on December 5, 2025

#316

Post by theplatman »

vlovich123 wrote: post #171
paradite wrote: post #31

The deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime lon…

That is also true at Cloudflare for what it’s worth. However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release, especially since there’s a 5 min lag (if I recall correctly) in the monitoring dashboards to get all the telemetry from thousands of servers worldwide. Comparing the difficulty of running the world’s internet traffic w…

With all due respect, engineers in finance can’t allow for outages like this because then you are losing massive amounts of money and potentially going out of business.

Re: Cloudflare outage on December 5, 2025

#317

Post by ectospheno »

agentifysh wrote: post #232
liampulles wrote: post #90

The lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase. The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.

yet my dedicated server has been up since 2015 with zero downtimes i dont think this is an entropy issue its human error bubbling up and cloudflare charges a premium for it my faith in cloudflare is shoook for sure two major outages weeks apart ad this wont be the last

Which 2015 kernel are you running?

Re: Cloudflare outage on December 5, 2025

#318

Post by XCSme »

al_borland wrote: post #311
XCSme wrote: post #295

Earlier quoted context omitted.

I mean, I still prefer to have the ability to fix it myself, because I know I can probably do it in 1h max. I know this doesn't apply to most people, especially those outside of HN though.

Even if resolution times are equal, there is some comfort in being able to see the problem and make progress on it to feel like you're actively doing something. I work in a large enterprise and we have a team dedicated to managing critical incidents and getting everyone together for a resolution. When a 3rd party vendor is the reason for the outage, those calls are really awkward. It's a bunch of people sitting aroun…

Today a client is having some issue with Zoom because of some artificial rate limits they impose. Their support is not responding, the account can't be used, courses can not be held and there's not much we can do.

We already started looking into moving away from Zoom, I suggested self-hosting http://jitsi.org Based on their docs, self-hosting is well supported, and probably a $50-$100 server is more than enough, so a lot cheaper than Zoom.

Post reply on HN