Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

421–430 of 601 posts

Re: Cloudflare outage on December 5, 2025

#421

Earlier quoted context omitted.

Usually I am allergic to pithy, vaguely dogmatic summaries like this but you're right. We have traded "some sites are down some of the time" for "most sites are down some of the time". Sure the "some" is eliding an order of magnitude or two, but this framing remains directionally correct.

Does relying on larger players result in better overall uptime for smaller players? AWS is providing me better uptime than if I assembled something myself because I am less resourced and less talented than that massive team. If so, is it a good or bad trade to have more overall uptime but when things go down it all goes down together?

> AWS is providing me better uptime than if I assembled something myself because I am less resourced and less talented than that massive team.

Is it? I can’t say that my personal server has been (unplanned) down at any time in the past 10 years, and these global outages have just flown right past it.

Re: Cloudflare outage on December 5, 2025

#422

Earlier quoted context omitted.

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

No you cannot! I blocked all of the user agents on a community wiki I run, and the traffic came back hours later masquerading as Firefox and Chrome. They just fucking lie to you and continue vacuuming your CPU.

There shouldn't be any noticeable hit on your cpu from bots from a site like that. Are you sure it's not a DDoS?

Obviously it depends on the bot, and you can't block the scammy ones. I was really just referring to the major legitimate companies (which might not include Perplexity).

Re: Cloudflare outage on December 5, 2025

#423
post #347

Earlier quoted context omitted.

Artifical limits, because they have 40 paid licenses that they can not use, because of a non-disclosed assignment limit that is NOT mentioned in the pricing page nor in the ToS. A lot of people are angry about this, and I think it's borderline illegal: https://devforum.zoom.us/t/you-have-exceeded-the-limit-of-li... You pay for something, and you can't use it.

If a company doesn't respond to this it tells you they likely only respond to lawsuits. As a paying customer whose business operations are impacted, you should have standing to sue. Your company could potentially extract from Zoom the entirety of the money that their dumb decision made your company lose. Consult a lawyer for actual advice and next steps. Of course, it's also possible you signed a contract that basica…

Probably not worth the effort, for a couple days of downtime, we'll just move somewhere else.

But I agree, I recognize the silence in that forum thread that was locked without a resolution: some boss said "let they complain or pay, we don't care about them otherwise".

Re: Cloudflare outage on December 5, 2025

#424

Earlier quoted context omitted.

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

How are you this naive? Do you really think scrapers give a damn about your robots.txt?

The legitimate ones do, which is what I was referring to. Obviously there are bastard ones as well.

Re: Cloudflare outage on December 5, 2025

#425

Earlier quoted context omitted.

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

This is just using robots.txt and asking "pretty please, don’t scrape me". Here is an article (from TODAY) about the case where Perplexity is being accused of ignoring robots.txt: https://www.theverge.com/news/839006/new-york-times-perplexi... If you think a robots.txt is the answer to stopping the billion-dollar AI machine from scraping you, I don’t know what to say.

Yes, I was referring to legitimate companies, and Perplexity doesn't seem to be one of those.

Re: Cloudflare outage on December 5, 2025

#426

Earlier quoted context omitted.

> Warning signs like this are how you know that something might be wrong! Yes, as they explain it's the rollback that was triggered due to seeing these errors that broke stuff.

They saw errors and decided to do a second rollout to disable the component generating errors, causing a major outage.

That was their first mistake, if your deployment does not behave the way you expect to (or even give you bad smell) roll back, that how it used to be... when I was a kid...lol Or I don't know, maybe load test before you deploy.....?

Re: Cloudflare outage on December 5, 2025

#427

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

I’m not sure I share this sentiment. First, let’s set aside the separate question of whether monopolies are bad. They are not good but that’s not the issue here. As to architecture: Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. But there’s a more interesting argume…

All of my company's hosted web sites have way better uptimes and availability than CF but we are utterly tiny in comparison.

With only some mild blushing, you could describe us as "artisanal" compared to the industrial monstrosities, such as Cloudflare.

Time and time again we get these sorts of issues with the massive cloudy chonks and they are largely due to the sort of tribalism that used to be enshrined in the phrase: "no one ever got fired for buying IBM".

We see the dash to the cloud and the shoddy state of in house corporate IT as a result. "We don't need in-house knowledge, we have "MS copilot 365 office thing" that looks after itself and now its intelligent - yay \o/

Until I can't, I'm keeping it as artisanal as I can for me and my customers.

Re: Cloudflare outage on December 5, 2025

#428

Earlier quoted context omitted.

I'm guessing you don't manage any production web servers? robots.txt isn't even respected by all of the American companies. Chinese ones (which often also use what are essentially botnets in Latin American and the rest of the world to evade detection) certainly don't care about anything short of dropping their packets.

I have been managing production commercial web servers for 28 years. Yes, there are various bots, and some of the large US companies such as Perplexity do indeed seem to be ignoring robots.txt. Is that a problem? It's certainly not a problem with cpu or network bandwidth (it's very minimal). Yes, it may be an issue if you are concerned with scraping (which I'm not). Cloudflare's "solution" is a much bigger problem th…

It is rather disingenuous to backpedal from "you can easily block them" to "is that a problem? who even cares" when someone points out that you cannot in fact easily block them.

Re: Cloudflare outage on December 5, 2025

#429

Earlier quoted context omitted.

No you cannot! I blocked all of the user agents on a community wiki I run, and the traffic came back hours later masquerading as Firefox and Chrome. They just fucking lie to you and continue vacuuming your CPU.

There shouldn't be any noticeable hit on your cpu from bots from a site like that. Are you sure it's not a DDoS? Obviously it depends on the bot, and you can't block the scammy ones. I was really just referring to the major legitimate companies (which might not include Perplexity).

There is a noticeable hit, there's also a noticeable cost, and it's not a ddos.

Not all sites can have full caching, we've tried.

Re: Cloudflare outage on December 5, 2025

#430
There's a lot of bad karma in this discussion. It's hard to run large services. Careful when you set a precedent of pillorying after an outage. It could be you next!

Yes, this is the second time in a month. Were folks expecting that to have been enough time for them to have made sweeping technical and organization changes? I say no—this doesn't mean they aren't trying or haven't learned any lessons from the last outage. It's a bit too soon to say that.

I see this event primarily as another example of the #1 class of major outages: bad rapid global configuration change. (The last CloudFlare outage was too, but I'm not just talking about CloudFlare. Google has had many many such outages. There was an inexplicable multi-year gap between recognizing this and having a good, widely available staged config rollout system for teams to drop into their systems.) Stuff like DoS attack configurations needs to roll out globally quickly. But they really need make it not quite this quick. Imagine they deployed to one server for one minute, one region for one minute on success, then everywhere on success. Then this would have been a tiny blip rather than a huge deal.

(It can be a bit hard to define "success" when you're doing something like blocking bad requests that may even be a majority of traffic during a DDoS attack, but noticing 100% 5xx errors for 38% of your users due to a parsing bug is doable!)

As for the specific bug: meh. They should have had 100% branch coverage on something as critical (and likely small) as the parsing for this config. Arguably a statically typed language would have helped (but the `.unwrap()` error in the previous outage is a bit of a counterargument to that). But it just wouldn't have mattered that much if they caught it before global rollout.

Post reply on HN