Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

501–510 of 601 posts

Re: Cloudflare outage on December 5, 2025

#501

Post by philipallstar »

Lamprey wrote: post #432
3rodents wrote: post #403

Earlier quoted context omitted.

Would you rather be attacked by 1,000 wasps or 1 dog? A thousand paper cuts or one light stabbing? Global outages are bad but the choice isn’t global pain vs local pleasure. Local and global both bring pain, with different, complicated tradeoffs. Cloudflare is down and hundreds of well paid engineers spring into action to resolve the issue. Your server goes down and you can’t get ahold of your Server Person because t…

It's not "1,000 wasps or 1 dog", it's "1,000 dogs at once, or "1 dog at once, 1,000 different times". Rare but huge and coordinated siege, or a steady and predictable background radiation of small issues. The latter is easier to handle, easier to fix, and much more suvivable if you do fuck it up a bit. It gives you some leeway to learn from mistakes. If you make a mistake during the 1000 dog siege, or if you don't ha…

I don't quite see how this maps onto the situation. The "1000 dog seige" also was resolved very quickly and transparently, so I would say it's actually better than even one of the "1 dog at once"s.

Re: Cloudflare outage on December 5, 2025

#502

Post by eviks »

> This first change was being rolled out using our gradual deployment system.

So they are aware of some basic mitigation tactics guarding against errors

> This system does not perform gradual rollouts,

They just choose to YOLO

> Typical actions are “block”, “log”, or “skip”. Another type of action is “execute”,

> However, we have never before applied a killswitch to a rule with an action of “execute”.

Do they do no testing? These isn't even fuzzing with “infinite” variations, but a limited list of actions

> existed undetected for many years. This type of code error is prevented by languages with strong type systems.

So this solution is also well known, just ignored for years, because "if it’s not broken, don’t fix it?", right?

Re: Cloudflare outage on December 5, 2025

#503

Post by vlovich123 »

autoexec wrote: post #225
vlovich123 wrote: post #217

Earlier quoted context omitted.

Can you name a major cloud provider that doesn’t have major outages? If this were purely a money problem it would have been solved ages ago. It’s a difficult problem to solve. Also, they’re the youngest of the major cloud providers and have a fraction of the resources that Google, Amazon, and Microsoft have.

> Can you name a major cloud provider that doesn’t have major outages? That fact that no major cloud provider is actually good is not an argument that cloudflare isn't bad, or even that they couldn't/shouldn't do better than they are. They have fewer resources than Google or Microsoft but they're also in a unique position that makes us differently vulnerable when they fuck up. It's not all their fault, since it was a…

Every major cloud provider (including Cloudflare) is orders of magnitude better at keeping 9s of availability worldwide for thousands of customers than those customers are individually. The very best of those customers might be better and only rely on cloud providers for the scaling or huge amounts of infrastructure they don’t otherwise want to own, but the vast majority are actually less capable at accomplishing whatever uptime the providers already get.

Could cloudflare do better? Sure, that’s a truism for everyone. Did they make mistakes and continue to make mistakes? Also a truism.

Trust me, they are acutely aware of people getting upset when they fail. Why do you think they’re CEO and CTO are writing these blog posts?

Re: Cloudflare outage on December 5, 2025

#504

Post by eviks »

uyzstvqs wrote: post #122

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately roll…

> Languages with strong type systems won't save you, good procedure will.

One of the items in the list of procedures is to use types to encode rules of your system.

Re: Cloudflare outage on December 5, 2025

#506

Post by locknitpicker »

toomuchtodo wrote: post #263
locknitpicker wrote: post #242

Earlier quoted context omitted.

> Do I want to deal with what Cloudflare offers? I do not, I have higher value work to focus on. I want to pay someone else to deal with this, and just like when cloud providers are down, it'll be back up eventually. This is specious reasoning. How come I had to endure a total outage due to the rollout of a mitigation of a Nextjs vulnerability when my organization doesn't even own any React app, let alone a Nextjs on…

If you are a customer of Cloudflare, and not happy, I encourage you to evaluate other providers more to your liking. Perhaps you'll find someone more fitting to your use case and operational preferences, but perhaps not. My day job org pays Cloudflare hundreds of thousands of dollars a year, and am satisfied with how they operate. Everyone has choice, exercise it if you choose. I'm sure your account exec would be hap…

> you are a customer of Cloudflare, and not happy, I encourage you to evaluate other providers more to your liking.

I think your take is terribly simplistic. In a professional setting, virtually all engineers have no say on whether the company switches platforms or providers. Their responsibility is to maintain and develop services that support business. The call to switch a provider is ultimately a business and strategic call, and is a subject that has extremely high inertia. You hired people specialized in technologies, and now you're just dumping all that investment? Not to mention contracts. Think about the problem this creates.

Some of you sound like amateurs toying with pet projects, where today it's framework A on cloud provider X whereas tomorrow it's framework B on cloud provider Y. Come the next day, rinse and repeat. This is unthinkable in any remotely professional setting.

Re: Cloudflare outage on December 5, 2025

#507

Post by bostik »

dotancohen wrote: post #353
bostik wrote: post #327

Earlier quoted context omitted.

> 2 minutes for their automated alerts to fire is terrible I take exception to that, to be honest. It's not desirable or ideal, but calling it "terrible" is a bit ... well, sorry to use the word ... entitled. For context, I have experience running a betting exchange. A system where it's common for a notable fraction of transactions in a medium-volume event to take place within a window of less than 30 seconds. Vast m…

Prometheus compatible but more reliable? Sell it to me!

VictoriaMetrics. The answer to the question "could I get Prometheus, but with ClickHouse architecture?"

Re: Cloudflare outage on December 5, 2025

#508

Post by atmosx »

tobyjsullivan wrote: post #396
mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

I’m not sure I share this sentiment. First, let’s set aside the separate question of whether monopolies are bad. They are not good but that’s not the issue here. As to architecture: Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. But there’s a more interesting argume…

CloudFlare doesn’t have a good track record. It’s the third party that caused more outages for us than any other third party service in the last four years.

Re: Cloudflare outage on December 5, 2025

#509

Post by raincole »

ccakes wrote: post #450
tobyjsullivan wrote: post #396

Earlier quoted context omitted.

I’m not sure I share this sentiment. First, let’s set aside the separate question of whether monopolies are bad. They are not good but that’s not the issue here. As to architecture: Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. But there’s a more interesting argume…

> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. The point is that it doesn’t matter. A single site going down has a very small chance of impacting a large number of users. Cloudflare going down breaks an appreciable portion of the internet. If Jim’s Big Blog only maintains 95% uptime, most people won’t care. If BofA were at 95%.. actually same. Most of the…

> A single site going down has a very small chance of impacting a large number of users

How? If Github is down how many people are affected? Google?

> Jim’s Big Blog only maintains 95% uptime, most people won’t care

Yeah, and in the world with Cloudflare people don't care if Jim's Blog is down either. So Cloudflare doesn't make things worse.

Re: Cloudflare outage on December 5, 2025

#510

Post by locknitpicker »

shermantanktop wrote: post #474
ccakes wrote: post #450

Earlier quoted context omitted.

> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. The point is that it doesn’t matter. A single site going down has a very small chance of impacting a large number of users. Cloudflare going down breaks an appreciable portion of the internet. If Jim’s Big Blog only maintains 95% uptime, most people won’t care. If BofA were at 95%.. actually same. Most of the…

Maybe worlds can just live without the internet for a few hours. There are likely emergency services dependent on Cloudflare at this point, so I’m only semi serious.

> Maybe worlds can just live without the internet for a few hours.

The world can also live a few hours without sewers, water supply, food, cars, air travel, etc.

But "can" and "should" are different words.

Post reply on HN