Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

591–600 of 601 posts

Re: Cloudflare outage on December 5, 2025

#591

Post by gerdesj »

foobarkey wrote: post #557
gerdesj wrote: post #427

Earlier quoted context omitted.

All of my company's hosted web sites have way better uptimes and availability than CF but we are utterly tiny in comparison. With only some mild blushing, you could describe us as "artisanal" compared to the industrial monstrosities, such as Cloudflare. Time and time again we get these sorts of issues with the massive cloudy chonks and they are largely due to the sort of tribalism that used to be enshrined in the phr…

Sorry for the downvotes but this is true many times with some basic HA you get better uptime than the big cloud boys, yes their stack and tech is fancier but we also need to factor in how much CF messes with it vs self hosted, anyway the self hosted wisdom is RIP these days and I mostly just run cf pages / kv :)

That sounds a bit complicated and rather weird.

I still have my webby assets up with way longer uptimes than the biggies.

I don't need internet points. A satisfied PO is what I want to see as a business owner.

Re: Cloudflare outage on December 5, 2025

#592

Post by fallous »

locknitpicker wrote: post #585
fallous wrote: post #559

Earlier quoted context omitted.

Your electricity to servers IS a single point of failure, if all you do is depend upon the power company to reliably feed power. There is a reason that co-location centers have UPS and generator backups for power. It may have been unthinkable to some casual observers that creating a giant single point of failure for the internet was a bad idea but it was entirely thinkable to others.

> Your electricity to servers IS a single point of failure, if all you do is depend upon the power company to reliably feed power. I think you quite didn't got the point. The whole point is that putting together a system architecture that considers Cloudflare is a single point of failure is like designing a system architecture that considers a power supplier a single point of failure. Technically they can be consider…

You're arguing from an end-user perspective, I'm pointing out that the Internet wasn't designed to solve easy but fragile problems but instead was intended to be a resilient network capable of surviving failures and route around them.

"I want to use a power tool and simply plug it into a wall" is not the same class of problem as "we're using a heart-lung machine during this bypass operation and power loss results in dead patients."

The widespread dependence upon Cloudflare has resulted in the "heart-lung machine" problem of DNS, among other things, being "solved" by a "power tool" class of solution.

Re: Cloudflare outage on December 5, 2025

#593

Post by lionkor »

ivanjermakov wrote: post #387
hashstring wrote: post #373

Earlier quoted context omitted.

Where/how are you keeping track of this? What is their current uptime percentage?

1 - downtime/period. I suspect period is 1 year. 99.9% is 8.76 hours of downtime a year.

Exactly this. Just keep a note every time they have a big outage, and then take the last year and see if its over 9 hours, is a good measure.

Re: Cloudflare outage on December 5, 2025

#594

Post by lionkor »

chickensong wrote: post #320
lionkor wrote: post #101

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

That's a pretty silly comparison though.

Yes, but it should also never happen that CF drops below 99.99, that's extremely doable for them (and it is literally the point of using a CDN).

Re: Cloudflare outage on December 5, 2025

#595

Post by 0x9e3779b6 »

Apart from Cloudflare config system working too good to propagate failure modes:

the code quality on very mission critical path powering “half the internets” could’ve been better.

I’m not sure if Lua LSP / linting tools would’ve caught the issue (I also never used Lua myself), but tools and methods exist to test mission critical dynamically typed code.

The company with genuinely impressive concentration of talent was expected to think about fuzzing this legacy crap somehow.

As for `.unwrap()` related incident: normally code like this should never pass the review.

You just (almost) never unwrap in production code.

I’d start with code quality tooling but more important - the related processes before even thinking about the architecture changes.

Changing architecture in global sense which, in general has served for years with 99.99(9)% uptime is not obviously smart thing to do.

Architecture is doing great, it’s just impact which has been devastating because of the scale.

Everyone makes errors and it’s fine, but there are ways not to roll shit in prod (reference to famous meme pic where bugs do that).

Re: Cloudflare outage on December 5, 2025

#596

Post by locknitpicker »

fallous wrote: post #592
locknitpicker wrote: post #585

Earlier quoted context omitted.

> Your electricity to servers IS a single point of failure, if all you do is depend upon the power company to reliably feed power. I think you quite didn't got the point. The whole point is that putting together a system architecture that considers Cloudflare is a single point of failure is like designing a system architecture that considers a power supplier a single point of failure. Technically they can be consider…

You're arguing from an end-user perspective, I'm pointing out that the Internet wasn't designed to solve easy but fragile problems but instead was intended to be a resilient network capable of surviving failures and route around them. "I want to use a power tool and simply plug it into a wall" is not the same class of problem as "we're using a heart-lung machine during this bypass operation and power loss results in…

> You're arguing from an end-user perspective,

No.I am arguing from a software engineer's perspective tackling a systems design problem.

> I'm pointing out that the Internet wasn't designed to solve easy but fragile problems but instead was intended to be a resilient network capable of surviving failures and route around them.

Irrelevant. Engineers design systems that remain functioning in spite of their failure modes. Some failure modes are irredeemable. Even structural engineers don't design structures to withstand all conceivable earthquakes, because they understand that mitigating that failure modes is unrealistic.

The same goes for software. You do not build your WebApps to remain working when half of the internet dies. This means scenarios such as AWS, GCP or Cloudflare being out.

Re: Cloudflare outage on December 5, 2025

#597

Post by smsm42 »

tobyjsullivan wrote: post #396
mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

I’m not sure I share this sentiment. First, let’s set aside the separate question of whether monopolies are bad. They are not good but that’s not the issue here. As to architecture: Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. But there’s a more interesting argume…

> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so.

That's a wrong way of looking at it though. For 99.99% individual sites, I wouldn't care if they were down for weeks. Even if I use this site, there are very few sites that I need to use daily. For the rest of them, if one of them randomly goes down I probably would never know or notice, because I didn't need it then. However, when single-point-of-failure provider, like Cloudflare, goes down, you bet I notice. I must notice, because my work would be affected, my CI/CD pipelines will start failing, my newsfeeds will stop, I will notice it in dozens of places - because everybody uses it. The aggregated fails-per-unit-of-time may be less but the impact of each fail is way, way more, and the probability of it impacting me is approaching certainty.

So for me, as an average internet user, it would be much better if all the world wouldn't go down at once, even if the instances of particular things going down would be more frequent - provided they are randomly distributed in time and not concentrated. If just one thing goes down, I could do another thing. If everything goes down, I can only sit and twiddle my thumbs until it's back up.

Re: Cloudflare outage on December 5, 2025

#599

Post by philipallstar »

rolisz wrote: post #541
philipallstar wrote: post #501

Earlier quoted context omitted.

I don't quite see how this maps onto the situation. The "1000 dog seige" also was resolved very quickly and transparently, so I would say it's actually better than even one of the "1 dog at once"s.

Last week's cloudflare outage was not resolved as quickly...

Perhaps not, but still compared to running your own thing you just sit and wait. No on call rotas to manage and pay for; no root cause analysis meetings after that descend into internal blame.

Re: Cloudflare outage on December 5, 2025

#600

Post by roguecoder »

nish__ wrote: post #119

Is it crazy to anyone else that they deploy every 5 minutes? And that it's not just config updates, but actual code changes with this "execute" action.

No: I've been at plenty of places where we get to continuous deployment, where any given change is deployed on demand.

What is wild is that they are deploying without first testing in a staging environment.

Post reply on HN