Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

381–390 of 601 posts

Re: Cloudflare outage on December 5, 2025

#381

Earlier quoted context omitted.

Akamai manages it.

They don't, akamai has had several outages as well jsut no one notices. Akamai is way way smaller than cloudflare, 20% of internet traffic passes through CF networks, not sure it's even measurable on Akamai.

Quickly Googling about, a commonly repeated figure is that Akamai served 15% - 30% of Internet traffic in the late 2010's. They probably have less of the market today due to others growing, but they're not a minnow.

2024 revenue figures were $1.669 billion for Cloudflare, and $3.99 billion for Akamai, per Wikipedia.

Re: Cloudflare outage on December 5, 2025

#382
post #327

Earlier quoted context omitted.

> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions. Their timeline: > 08:47: Configuration change deplo…

> 2 minutes for their automated alerts to fire is terrible I take exception to that, to be honest. It's not desirable or ideal, but calling it "terrible" is a bit ... well, sorry to use the word ... entitled. For context, I have experience running a betting exchange. A system where it's common for a notable fraction of transactions in a medium-volume event to take place within a window of less than 30 seconds. Vast m…

Critical high-level stats such as errors should be scraped more frequently than 30 seconds. It’s important to have multiple time granularity scraping intervals, a small set of most critical stats should be scraped closer to 10s or 15s.

Prometheus has as an unaddressed flaw [0], where rate functions must be at least 2x the scrape interval. This means that if you scrape at 30s intervals, your rate charts won’t reflect the change until a minute after.

[0] - https://github.com/prometheus/prometheus/issues/3746

Re: Cloudflare outage on December 5, 2025

#383

Earlier quoted context omitted.

> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions. Their timeline: > 08:47: Configuration change deplo…

I see lots of people complaining about this down time but in actuality is it really that big a deal to have 30 minutes of down time or whatever. It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake. In many developed countries the electric power service has local down times on occasion. That's more important than not being able…

I am confident there is at least a few hospitals, gp offices or ticketing systems that interact directly or indirectly with Cloud flare. They've sold themselves as a major defence in security.

Re: Cloudflare outage on December 5, 2025

#384

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

Robust architecture that is serving 80M requests/second worldwide?

My answer would be that no one product should get this big.

Re: Cloudflare outage on December 5, 2025

#385

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

You're not wrong, but where's the robust architecture you're referring to? The reality of providing reliable services on the internet is far beyond the capabilities of most organizations.

Re: Cloudflare outage on December 5, 2025

#386
post #369

Earlier quoted context omitted.

In other words, the consolidation on Cloudflare and AWS makes the web less stable. I agree.

Usually I am allergic to pithy, vaguely dogmatic summaries like this but you're right. We have traded "some sites are down some of the time" for "most sites are down some of the time". Sure the "some" is eliding an order of magnitude or two, but this framing remains directionally correct.

Does relying on larger players result in better overall uptime for smaller players? AWS is providing me better uptime than if I assembled something myself because I am less resourced and less talented than that massive team.

If so, is it a good or bad trade to have more overall uptime but when things go down it all goes down together?

Re: Cloudflare outage on December 5, 2025

#387

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

Where/how are you keeping track of this? What is their current uptime percentage?

1 - downtime/period. I suspect period is 1 year. 99.9% is 8.76 hours of downtime a year.

Re: Cloudflare outage on December 5, 2025

#388

Earlier quoted context omitted.

Management doesn't like when things like this are automated. They want to "manage" the outage/production/etc numbers before letting them out.

There's no sweet spot I've found. I don't work for Cloudflare but when I did have a status indicator to maintain, you could never please everyone. Users would complain when our system was up but a dependent system was down, saying that our status indicator was a lie. "Fixing" that by marking our system as down or degraded whenever a dependent system was down led to the status indicator being not green regularly, caus…

> "Fixing" that by marking our system as down or degraded whenever a dependent system was down led to the status indicator being not green regularly, causing us to unfairly develop a reputation as unreliable (most broken dependencies had limited blast radius).

This seems like an issue with the design of your status page. If the broken dependencies truly had a limited blast radius, that should've been able to be communicated in your indicators and statistics. If not, then the unreliable reputation was deserved, and all you did by removing the status page was hide it.

Re: Cloudflare outage on December 5, 2025

#389

Earlier quoted context omitted.

Just try harder. And if it still breaks, clearly you weren't trying hard enough! At some point you have to admit that humans are pretty bad at some things. Keeping documentation up to date and coherent is one of those things, especially in the age of TikTok. Better to live in the world we have and do the best you can, than to endlessly argue about how things should be but never will become.

> especially in the age of TikTok Shouldn't grey beards, grizzled by years of practicing rigorous engineering, be passing this knowledge on to the next generation? How did they learn it when just starting out? They weren't born with it. Maybe engineering has actually improved so much that we only need to experience outages this frequently, and such feelings of nostalgia are born from never having to deal with systems…

We were talking about making a missile (v2) with an extended range, and ensuring that the developers who work on it understand the assumption of the prior model: that it doesn't use free because it's expected to blow up before that would become an issue (a perfectly valid approach, I might add). And to ensure that this assumption still holds in the v2 extended range model. The analogy to Ariane 5 is very apt.

Now, there can be tens of thousands of similar considerations to document. And keeping up that documentation with the actual state of the world is a full time job in itself.

You can argue all you want that folks "should" do this or that, but all I've seen in my entire career is that documentation is almost universally: out of date, and not worth relying on because it's actively steering you in the wrong direction. And I actually disagree (as someone with some gray in my beard) with your premise that this is part of "rigorous engineering" as is practiced today. I wish it was, but the reality is you have to read the code, read it again, see what it does on your desk, see what it does in the wild, and still not trust it.

We "should" be nice to each other, I "should" make more money, and it "should" be sunny more often. And we "should" have well written, accurate and reliable docs, but I'm too old to be waiting around for that day to come, especially in the age of zero attention and AI generated shite.

Re: Cloudflare outage on December 5, 2025

#390

Earlier quoted context omitted.

You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? I would say Cloudflare is giving these site owners an option to protect their content and as a byproduct, reduce their own costs of subsidizing their thieves. They can choose to turn off the crawl protection. If they aren't, that tells you that they want it, doesn’t it?

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

Tell me you don't run a site without telling me you don't run a site
Post reply on HN