Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

491–500 of 601 posts

Re: Cloudflare outage on December 5, 2025

#491
From a customer perspective, I think there should be an option:

- prioritize security: get patchs ASAP

- prioritize availability: get patchs after a cooldown period

Because ultimately, it's a tradeoff that cannot be handled by Cloudflare. It depends on your business, your threat model.

Re: Cloudflare outage on December 5, 2025

#492

Earlier quoted context omitted.

That's an interesting point, but in many (most?) cases productivity doesn't depend on all services being available at the same time. If one service goes down, you can usually be productive by using an alternative (e.g. if HN is down you go to Reddit, if email isn't working you catch up with Slack).

If HN, Reddit, email, Slack and everything else is down for a day, I think my productivity would actually go up, not down.

During 1st Cloudflare outage StackOverflow was down too.

Re: Cloudflare outage on December 5, 2025

#493
post #450

Earlier quoted context omitted.

> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. The point is that it doesn’t matter. A single site going down has a very small chance of impacting a large number of users. Cloudflare going down breaks an appreciable portion of the internet. If Jim’s Big Blog only maintains 95% uptime, most people won’t care. If BofA were at 95%.. actually same. Most of the…

Look at it a user (or even operator) of one individual service that isn’t redundant or safety critical: if choice A has 1/2 the downtime of choice B, you can’t justify choosing choice B by virtue of choice A’s instability.

That is exactly why you don't see Windows being used anymore in big corporations. /s

Re: Cloudflare outage on December 5, 2025

#494

I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI exto…

the two things are unrelated...

The pay-per-crawl thing, is about them thinking ahead about post-AI business/revenue models.

The way AI happened, it removed a big chunk of revenue from news companies, blogs, etc. Because lots of people go to AI instead of reaching the actual 3rd party website.

AI currently gets the content for free from the 3rd party websites, but they have revenue from their users.

So Cloudflare is proposing that AI companies should be paying for their crawling. Cloudflare's solution would give the lost revenue back where it belongs, just through a different mechanism.

The ugly side of the story is that this was already an existing solution, and open source, called L402.org.

Cloudflare wants to be the first to take a piece of the pie, but also instead of using the open source version, they forked it internally and published it as their own service, which is cloudflare specific.

To be completely fair, the l402 requires you to solve the payment mechanism itself, which for Cloudflare is easy because they already deal with payments.

Re: Cloudflare outage on December 5, 2025

#495
post #166

Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineeri…

It's weird reading these reports because they don't seem to test anything at all (or at least there's very little mention of testing).

Canary deployment, testing environments, unit tests, integration tests, anything really?

It sounds like they test by merging directly to production but surely they don't

Re: Cloudflare outage on December 5, 2025

#496

The interesting part: After rolling out a bad ruleset update, they tried a killswitch (rolled out immediately to 100%) which was a code path never executed before: > However, we have never before applied a killswitch to a rule with an action of “execute”. When the killswitch was applied, the code correctly skipped the evaluation of the execute action, and didn’t evaluate the sub-ruleset pointed to by it. However, an…

> have never before applied a killswitch to a rule with an action of “execute” One might think a company on the scale of Cloudflare would have a suite of comprehensive tests to cover various scenarios.

Yeah the example they gave does feel like pretty isolated unit test territory, or at least an integration test on a subset of the system that could be ran in isolation.

Re: Cloudflare outage on December 5, 2025

#497

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

Yeah, redundancy and efficiency are opposites. As engineers, we always chase efficiency, but resilience and redundancy are related.

Re: Cloudflare outage on December 5, 2025

#498

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately roll…

I am sure they have this. What tends to happen is that the gradual rollout system becomes too slow for some rare, low latency rollout requirements, so a config system is introduced that fulfills the requirements. For example, let’s say you have a gradual rollout for binaries (slow) and configuration (fast). Over time, the fast rollout of the configuration system will cause outages, so it’s slowed down. Then a requirement pops up for which the config system is too slow and someone identifies a global system with no gradual rollout (e.g. a database) to be used as the solution. That solution will be compliant with all the processes that have been introduced to the letter, because so far nobody has thought of using a single database row for global configuration yet. Add new processes whenever this happens and at some point everything will be too slow and taking on more risk becomes necessary to stay competitive. So processes are adjusted. Repeat forever.

Re: Cloudflare outage on December 5, 2025

#499

I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI exto…

In my experience it's been in recent years, not months.

Re: Cloudflare outage on December 5, 2025

#500

If I'm remembering correctly, there was another outage around 10 days ago. It still surprises me that there are basically no free alternatives comparable to Cloudflare. Putting everything on CF creates a pretty serious single point of failure. It's strange that in most industries you have at least two major players, like Coke vs. Pepsi or Nike vs. Adidas. But in the CDN/edge space, there doesn't seem to be a real fre…

I reckon AWS is "free enough" for most of its users, but it's not as easy nor as safe for the common user.

Totally agree, AWS's free tier is great for many users, but it can definitely be tricky and risky for the average person.
Post reply on HN