Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

211–220 of 601 posts

Re: Cloudflare outage on December 5, 2025

#211

I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI exto…

You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? I would say Cloudflare is giving these site owners an option to protect their content and as a byproduct, reduce their own costs of subsidizing their thieves. They can choose to turn off the crawl protection. If they aren't, that tells you that they want it, doesn’t it?

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious?

You can easily block ChatGPT and most other AI scrapers if you want:

https://habeasdata.neocities.org/ai-bots

Re: Cloudflare outage on December 5, 2025

#212
post #75

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

One thing to keep in mind when judging what's 'appropriate' is that Cloudflare was effectively responding to an ongoing security incident outside of their control (the React Server RCE vulnerability). Part of Cloudlfare's value proposition is being quick to react to such threats. That changes the equation a bit: any hour you wait longer to deploy, your customers are actively getting hacked through a known high-severi…

Cloudflare had already decided this was a rule that could be rolled out using their gradual deployment system. They did not view it as being so urgent that it required immediate global roll out.

Re: Cloudflare outage on December 5, 2025

#213

The deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime lon…

That is also true at Cloudflare for what it’s worth. However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release, especially since there’s a 5 min lag (if I recall correctly) in the monitoring dashboards to get all the telemetry from thousands of servers worldwide. Comparing the difficulty of running the world’s internet traffic w…

> However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release

This kind of thing would be more understandable for a company without hundreds of billions of dollars, and for one that hasn't centralized so much of the internet. If a company has grown too large and complex to be well managed and effective and it's starting to look like a liability for large numbers of people there are obvious solutions for that.

Re: Cloudflare outage on December 5, 2025

#214
post #16

Whats going on with cloudflare's software team? I have seen similar bugs in cloudflare API recently as well. There is an endpoint for a feature that is available only to enterprise users, but the check for whether the user is on an enterprise plan is done at the last step.

Can you elaborate? I'm not sure what you mean by "at the last step"

I'm not sure which endpoint gp meant, but as I understood it, as an example, imagine a three-way handshake that's only available to enterprise users. Instead of failing a regular user on the first step, they allow steps one and two, but then do the check on step three and fail there.

Re: Cloudflare outage on December 5, 2025

#216

The deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime lon…

Cloudflare is orders of magnitude larger than any fintech. Rollouts likely take much longer, and having a human monitoring a dashboard doesn't scale.

> Rollouts likely take much longer

Cloudflare’s own post says the configuration change that resulted in the outage rolled out in seconds.

Re: Cloudflare outage on December 5, 2025

#217

Earlier quoted context omitted.

That is also true at Cloudflare for what it’s worth. However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release, especially since there’s a 5 min lag (if I recall correctly) in the monitoring dashboards to get all the telemetry from thousands of servers worldwide. Comparing the difficulty of running the world’s internet traffic w…

> However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release This kind of thing would be more understandable for a company without hundreds of billions of dollars, and for one that hasn't centralized so much of the internet. If a company has grown too large and complex to be well managed and effective and it's starting to look li…

Can you name a major cloud provider that doesn’t have major outages?

If this were purely a money problem it would have been solved ages ago. It’s a difficult problem to solve. Also, they’re the youngest of the major cloud providers and have a fraction of the resources that Google, Amazon, and Microsoft have.

Re: Cloudflare outage on December 5, 2025

#219
post #188

Earlier quoted context omitted.

> doesn't really justify the decision to use the same deployment system responsible for the 11/18 outage There’s no other deployment system available. There’s a single system for config deployment and it’s all that was available as they haven’t yet done the progressive roll out implementation yet.

Ok. Sure But shouldn't they have some beta/staging/test area they could deploy to, run tests for an hour then do the global blast?

Config changes are distinctly more difficult to have that set up for and as the blog says they’re working on it. They just don’t have it ready yet and are pausing any more config changes until it’s set up. They just did this one in response to try to mitigate an ongoing security vulnerability and missed the mark.

I’m happy to see they’re changing their systems to fail open which is one of the things I mentioned in the conversation about their last outage.

Re: Cloudflare outage on December 5, 2025

#220

The lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase. The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.

I'm not sure how decentralization helps though. People in a bazzar are going to care even less about sharing shadow knowledge. Linux IMO succeeds not because of the bazaar but because of Linus.

You don't keep a bazaar running with shadow knowledge. Either the important things are published or it doesn't run.
Post reply on HN