Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

371–380 of 601 posts

Re: Cloudflare outage on December 5, 2025

#371

Post by runlaszlorun »

braiamp wrote: post #190
w10-1 wrote: post #166

Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineeri…

This is funny, considering that someone that worked on the defense industry (guide missile system) found a memory leak on one of their products, at that time. They told him that they knew about it, but that it's timed just right with the range of the system it would be used, so it doesn't matter.

My hunch is that we do the same with memory leaks or other bugs in web applications where the time of a request is short.

Re: Cloudflare outage on December 5, 2025

#372

Post by 8note »

Scaevolus wrote: post #6

> Disabling this was done using our global configuration system. This system does not use gradual rollouts but rather propagates changes within seconds to the entire network and is under review following the outage we recently experienced on November 18. > As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following LUA exception: They r…

they arent a panacea though, internal tools like that can be super noisy on errors, and be broken more often than theyre working

Re: Cloudflare outage on December 5, 2025

#374

Post by hashstring »

tripplyons wrote: post #258
lionkor wrote: post #101

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

Do they include uptime guarantees in any contracts?

See 1.1 and 1.2; you get some credit back for when they fail to deliver.

https://www.cloudflare.com/business-sla/

Re: Cloudflare outage on December 5, 2025

#375

Post by autoexec »

evanelias wrote: post #355
autoexec wrote: post #343

Earlier quoted context omitted.

That was admittedly hyperbole, but since we're talking about a company with assets and revenue in the billions I'm not sure it matters. The fact remains that a lack of money/resources is not their problem.

They don't have unlimited resources. They have ~5000 employees. That's not small but it's not huge either. For sake of comparison, Google hit that headcount level literally 20 years ago.

They have enough money to buy anything they need. The CEO alone has billions. He could pay for as many employees as he wants out of his own pocket and not notice. In fact he's good at buying people, even senators.

Re: Cloudflare outage on December 5, 2025

#376

Post by froober »

evanelias wrote: post #270
autoexec wrote: post #213

Earlier quoted context omitted.

> However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release This kind of thing would be more understandable for a company without hundreds of billions of dollars, and for one that hasn't centralized so much of the internet. If a company has grown too large and complex to be well managed and effective and it's starting to look li…

What "hundreds of billions of dollars"? Cloudflare's annual revenue is around $2 billion, and they are not yet profitable.

Given how well-established cloudflare is, I would've figured they'd be profitable by now. That raises the question: why does so much of the web rely on a company which does not have the means to sustain itself?

Re: Cloudflare outage on December 5, 2025

#377

Post by amazingman »

WD-42 wrote: post #369
mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

In other words, the consolidation on Cloudflare and AWS makes the web less stable. I agree.

Usually I am allergic to pithy, vaguely dogmatic summaries like this but you're right. We have traded "some sites are down some of the time" for "most sites are down some of the time". Sure the "some" is eliding an order of magnitude or two, but this framing remains directionally correct.

Re: Cloudflare outage on December 5, 2025

#378

Post by mrb »

jacobgkau wrote: post #288

I noticed this outage last night (Cloudflare 500s on a few unrelated websites). As usual, when I went to Cloudflare's status page, nothing about the outage was present; the only thing there was a notice about the pre-planned maintenance work they were doing for the security issue, reporting that everything was being routed around it successfully.

Only way to change that it to shame them for it: "Cloudflare is so incompetent at detecting and managing outages that even their simple status page is unable to be accurate"

If enough high-ranked customers report this feedback...

Re: Cloudflare outage on December 5, 2025

#379

Post by bdangubic »

froober wrote: post #376
evanelias wrote: post #270

Earlier quoted context omitted.

What "hundreds of billions of dollars"? Cloudflare's annual revenue is around $2 billion, and they are not yet profitable.

Given how well-established cloudflare is, I would've figured they'd be profitable by now. That raises the question: why does so much of the web rely on a company which does not have the means to sustain itself?

given how much of the population relies on Uber for their transportation… ;)

Re: Cloudflare outage on December 5, 2025

#380

Post by Etheryte »

wkat4242 wrote: post #370
colechristensen wrote: post #361

Earlier quoted context omitted.

It is not dysfunctional to ignore unreachable "bugs". A memory leak on a missile which won't be reached because it will explode long before that amount of time has passed is not a bug.

It's a debt though. Because people will forget it's there and then at some point someone changes a counter from milliseconds to microseconds and then the issue happens 1000 times sooner. It's never right to leave structural issues even if "they don't happen under normal conditions".

I don't think this argument makes sense. You wouldn't provision a 100GB server for a service where 1GB would do just in case unexpected conditions come up. If the requirements change, then the setup can change, doing it just because is wasteful. What if we forget is not a valid argument to over engineer and over provision.
Post reply on HN