So Cloudflare:
- Did a last minute, untested change to their change: "turning off our WAF rule testing tool".
- Did an immediate global roll-out, instead of a staged one.
.
Is seems they would have enough leaning-cases now never to do that again...
Cloudflare outage on December 5, 2025
531–540 of 601 posts
Re: Cloudflare outage on December 5, 2025
#532Earlier quoted context omitted.
Prometheus compatible but more reliable? Sell it to me!
VictoriaMetrics. The answer to the question "could I get Prometheus, but with ClickHouse architecture?"
I'll be looking into that, thank you.
While we're here, any other Prometheus or Grafana advice is welcome.
Re: Cloudflare outage on December 5, 2025
#533First, what Cloudflare does is hard and I want to start with that.
That being said, I think it’s worth a discussion. How much of the last 3 outages were because of the JGC (the former CTO) retiring and Dane taking over?
Did JGC have a steady hand that’s missing? Or was it just time for outages that would have happened anyway?
Dane has maintained a culture of transparency which is fantastic, but did something get injected in the culture leading towards these issues? Will it become more or less stable since JGC left?
Curious for anyone with some insight or opinions.
(Also, if it wasn’t clear - huge Cloudflare fan and sending lots of good vibes to the team)
Re: Cloudflare outage on December 5, 2025
#534This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…
What you've identified here is a core part of what the banking sector calls the "risk based approach". Risk in that case is defined as the product of the chance of something happening and the impact of it happening. With this understanding we can make the same argument you're making, a little more clearly.
Cloudflare is really good at what they do, they employ good engineering talent, and they understand the problem. That lowers the chance of anything bad happening. On the other hand, they achieve that by unifying the infrastructure for a large part of the internet, raising the impact.
The website operator herself might be worse at implementing and maintaining the system, which would raise the chance of an outage. Conversely, it would also only affect her website, lowering the impact.
I don't think there's anything to dispute in that description. The discussion then is if cloudflares good engineering lowers the chance of an outage happening more than it raises the impact. In other words, the things we can disagree about is the scaling factors, the core of the argument seems reasonable to me.
Re: Cloudflare outage on December 5, 2025
#535Earlier quoted context omitted.
They will remember, because it'll have been measured and documented, rigorously.
I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.
When people don't read the documentation, discovery is a real problem. When people do read the documentation, things are different. Many software engineers do not read the documentation, and then complain to you if they break something in a documented way. If you compare to hardware engineers, whose vendors put out tens of thousands of pages of documentation for single parts, they have a lot of skill at reading documentation (and the vendors at writing it).
Re: Cloudflare outage on December 5, 2025
#536Earlier quoted context omitted.
Does relying on larger players result in better overall uptime for smaller players? AWS is providing me better uptime than if I assembled something myself because I am less resourced and less talented than that massive team. If so, is it a good or bad trade to have more overall uptime but when things go down it all goes down together?
From a societal view it is worse when everything is down at once. Leads to a less resilient society: It is not great if I can't buy essentials from one store because their payment system is down (this happened to one super market chain in Sweden due to a hacker attack some years ago, took weeks to fully fix everything, and then there was that whole Crowdstrike debacle globally more recently). It is far worse if all o…
it's not that simple, no?
users want to do things, if their goal depends on a complex chain of functions (provided by various semi-independent services) then the ideal setup would be to have redundant providers and users could simply "load balance" between them and that separate high-level providers' uptime state is clustered (meaning that when Google is unavailable Bing is up, and when Random Site A, goes down their payment provider goes down too, etc..)
So ideally sites would somehow sort themselves nearly to separate availability groups.
Otherwise simply having a lot of uncorrelated downtimes doesn't help (if we count the sum of downtime experienced by people). Though again it gets complicated by the downtime percentage, because likely there's a phase shift between the states when user can mostly complete their goals and when they cannot because too many cascading failures.
Re: Cloudflare outage on December 5, 2025
#537Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.
When a piece of hardware goes or a careless backup process fails, downtime of a self-hosted service can be measured in days or weeks.
Have 2 of them and your users only see correlated failures.
Re: Cloudflare outage on December 5, 2025
#538Earlier quoted context omitted.
I’m not sure I share this sentiment. First, let’s set aside the separate question of whether monopolies are bad. They are not good but that’s not the issue here. As to architecture: Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. But there’s a more interesting argume…
> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. The point is that it doesn’t matter. A single site going down has a very small chance of impacting a large number of users. Cloudflare going down breaks an appreciable portion of the internet. If Jim’s Big Blog only maintains 95% uptime, most people won’t care. If BofA were at 95%.. actually same. Most of the…
I'm not sure I follow the argument. If literally every individual site had an uncorrelated 99% uptime, that's still less available than a centralized 99.9% uptime. The "entire Internet" is much less available in the former setup.
It's like saying that Chipotle having X% chance of tainted food is worse than local burrito places having 2*X% chance of tainted food. It's true in the lens that each individual event affects more people, but if you removed that Chipotle and replaced with all local, the total amount of illness is still strictly higher, it's just tons of small events that are harder to write news articles about.
Re: Cloudflare outage on December 5, 2025
#539Earlier quoted context omitted.
It's weird reading these reports because they don't seem to test anything at all (or at least there's very little mention of testing). Canary deployment, testing environments, unit tests, integration tests, anything really? It sounds like they test by merging directly to production but surely they don't
In the post they described that they observed errors happening in their testing env, but decided to ignore because they were rolling out a security fix. I am sure there is more nuance to this, but I don’t know whether that makes it better or worse
> but decided to ignore because they were rolling out a security fix.
A key part of secure systems is availability...
It really looks like vibe-coding.
Re: Cloudflare outage on December 5, 2025
#540Earlier quoted context omitted.
I’m not sure I share this sentiment. First, let’s set aside the separate question of whether monopolies are bad. They are not good but that’s not the issue here. As to architecture: Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. But there’s a more interesting argume…
That’s fine if it’s just some random office workers. What if every airline goes down at the same time because they all rely on the same backend providers? What if every power generator shuts off? “Everything goes down simultaneously” is not, in general, something to aim for.
That is literally how a large fraction of airlines work. It's called Amadeus, and it did have a big global outage not too long ago.