Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

411–420 of 601 posts

Re: Cloudflare outage on December 5, 2025

#411

Post by notepad0x90 »

jadamson wrote: post #328
notepad0x90 wrote: post #264

Earlier quoted context omitted.

You really should take some of your pill. > Instead, it was triggered by changes being made to our body parsing logic while attempting to detect and mitigate an industry-wide vulnerability disclosed this week in React Server Components. > Unfortunately, in our FL1 version of our proxy, under certain circumstances, the second change of turning off our WAF rule testing tool caused an error state that resulted in 500 HT…

> The body parsing logic is in react or nextjs, that's my takeaway, is it that incorrect? The exploit they were trying to protect against is in React services run by their customers.

that makes better sense now, thanks. I feel dumb now that I re-read it, in my mind they patched nextjs/react and the new patch somehow required more buffer size.

Re: Cloudflare outage on December 5, 2025

#412

Post by psunavy03 »

3rodents wrote: post #403
mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

Would you rather be attacked by 1,000 wasps or 1 dog? A thousand paper cuts or one light stabbing? Global outages are bad but the choice isn’t global pain vs local pleasure. Local and global both bring pain, with different, complicated tradeoffs. Cloudflare is down and hundreds of well paid engineers spring into action to resolve the issue. Your server goes down and you can’t get ahold of your Server Person because t…

If you've allowed your Server Person to be a single point of failure out innawoods, that's an organizational problem, not a technological one.

Two is one and one is none.

Re: Cloudflare outage on December 5, 2025

#413

Post by immibis »

XCSme wrote: post #347
carl_dr wrote: post #342

Earlier quoted context omitted.

Artifical limits because they are on the free plan?

Artifical limits, because they have 40 paid licenses that they can not use, because of a non-disclosed assignment limit that is NOT mentioned in the pricing page nor in the ToS. A lot of people are angry about this, and I think it's borderline illegal: https://devforum.zoom.us/t/you-have-exceeded-the-limit-of-li... You pay for something, and you can't use it.

If a company doesn't respond to this it tells you they likely only respond to lawsuits. As a paying customer whose business operations are impacted, you should have standing to sue. Your company could potentially extract from Zoom the entirety of the money that their dumb decision made your company lose. Consult a lawyer for actual advice and next steps.

Of course, it's also possible you signed a contract that basically says "we can just decide not to work and you can't do anything about it" in which case, sucks, and fire whoever negotiates your B2B contracts. But also, those clauses can be void if the violation is serious enough.

Re: Cloudflare outage on December 5, 2025

#414

Post by cpncrunch »

jacobgkau wrote: post #266
cpncrunch wrote: post #211

Earlier quoted context omitted.

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

I'm guessing you don't manage any production web servers? robots.txt isn't even respected by all of the American companies. Chinese ones (which often also use what are essentially botnets in Latin American and the rest of the world to evade detection) certainly don't care about anything short of dropping their packets.

I have been managing production commercial web servers for 28 years.

Yes, there are various bots, and some of the large US companies such as Perplexity do indeed seem to be ignoring robots.txt.

Is that a problem? It's certainly not a problem with cpu or network bandwidth (it's very minimal). Yes, it may be an issue if you are concerned with scraping (which I'm not).

Cloudflare's "solution" is a much bigger problem that affects me multiple times daily (as a user of sites that use it), and those sites don't seem to need protection against scraping.

Re: Cloudflare outage on December 5, 2025

#415

Post by morpheos137 »

odie5533 wrote: post #307
morpheos137 wrote: post #268

Earlier quoted context omitted.

I see lots of people complaining about this down time but in actuality is it really that big a deal to have 30 minutes of down time or whatever. It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake. In many developed countries the electric power service has local down times on occasion. That's more important than not being able…

> It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake. Yes, there are lots of mission critical systems that use cloudflare and lives and huge amounts of money are at stake.

Can you provide an example of lives being at stake because of a cloud flare outage?

Re: Cloudflare outage on December 5, 2025

#416

Post by NicoJuicy »

mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

You should really check Cloudflare.

There is not a single company that makes their infrastructure as globally available like Cloudflare.

Additionally, the downtime of Cloudflare seems to be objectively less than the others.

Now, it took 25 minutes for 28% of the network.

While being the only ones to fix a global vulnerability.

There is a reason other clouds wouldn't touch the responsiveness and innovation that Cloudflare brings.

Re: Cloudflare outage on December 5, 2025

#417

Post by JohnMakin »

lapcat wrote: post #200
JohnMakin wrote: post #196

Earlier quoted context omitted.

I am not changing any subject. These are glue logic scripts connecting massive pieces of infra together, spanning what is likely several teams and orgs over the course of many years. It is impossible to blurt something out like "well, source code testing" for something like this, when the source code inputs are not possibly testable outside the scale of the larger system. They're often completely unknowable as well.…

With all due respect, it sounds like you have never heard of unit tests. Cloudflare states that the compiler would prevent the bug in certain programming languages. So it seems ridiculous to suggest that the bug can't be detected outside the scale of a larger system.

Please explain how unit tests stop a problem from propagating across a system that fields 70 million requests a second and I’ll take you more seriously, otherwise I’m done with this particular subthread.

Re: Cloudflare outage on December 5, 2025

#418

Post by theideaofcoffee »

ilkkao wrote: post #42
gkoz wrote: post #26

I sometimes feel we'd be better off without all the paternalistic kitchensink features. The solid, properly engineered features used intentionally aren't causing these outages.

Agreed, I don't really like Cloudflare trying to magically fix every web exploit there is in frameworks my site has never used.

I’ve been downvoted enough with my comments on this blog post where I’m hesitant to add anything else, but here I agree with you. They’re trying to be everything to everyone, where does the accountability of their customers being responsible for running, you know, up-to-date packages come in? Like, you don’t take just a little bit of pride in your work that you’re continually watching CVE lists and exploits and just have a minimum of effort toward patching your own shit, rather than pawning it off on vendor? I simply can’t understand the mindset.

Re: Cloudflare outage on December 5, 2025

#419

Post by cpncrunch »

literalAardvark wrote: post #390
cpncrunch wrote: post #211

Earlier quoted context omitted.

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

Tell me you don't run a site without telling me you don't run a site

Tell me you make incorrect assumptions without specifically saying so. (Yes, you're incorrect).

Re: Cloudflare outage on December 5, 2025

#420

Post by Lamprey »

PullJosh wrote: post #386
amazingman wrote: post #377

Earlier quoted context omitted.

Usually I am allergic to pithy, vaguely dogmatic summaries like this but you're right. We have traded "some sites are down some of the time" for "most sites are down some of the time". Sure the "some" is eliding an order of magnitude or two, but this framing remains directionally correct.

Does relying on larger players result in better overall uptime for smaller players? AWS is providing me better uptime than if I assembled something myself because I am less resourced and less talented than that massive team. If so, is it a good or bad trade to have more overall uptime but when things go down it all goes down together?

When only one thing goes down, it's easier to compensate with something else, even for people who are doing critical work but who can't fix IT problems themselves. It means there are ways the non-technical workforce can figure out to keep working, even if the organization doesn't have on-site IT.

Also, if you need to switchover to backup systems for everything at once, then either the backup has to be the same for everything and very easily implementable remotely - which to me seems unlikely for specialty systems, like hospital systems, or for the old tech that so many organizations still rely on (and remember the CrowdStrike BSODs that had to be fixed individually and in person and so took forever to fix?) - or you're gonna need a LOT of well-trained IT people, paid to be on standby constantly, if you want to fix the problems quickly, on account of they can't be everywhere at once.

If the problems are more spread out over time, then you don't need to have quite so many IT people constantly on standby. Saves a lot of $$$, I'd think.

And if problems are smaller and more spread out over time, then an organization can learn how to deal with them regularly, as opposed to potentially beginning to feel and behave as though the problem will never actually happen. And if they DO fuck up their preparedness/response, the consequences are likely less severe.

Post reply on HN