Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

431–440 of 601 posts

Re: Cloudflare outage on December 5, 2025

#431

Post by rossjudson »

mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

You have a heterogeneous, fault-free architecture for the Cloudflare problem set? Interesting! Tell us more.

Re: Cloudflare outage on December 5, 2025

#432

Post by Lamprey »

3rodents wrote: post #403
mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

Would you rather be attacked by 1,000 wasps or 1 dog? A thousand paper cuts or one light stabbing? Global outages are bad but the choice isn’t global pain vs local pleasure. Local and global both bring pain, with different, complicated tradeoffs. Cloudflare is down and hundreds of well paid engineers spring into action to resolve the issue. Your server goes down and you can’t get ahold of your Server Person because t…

It's not "1,000 wasps or 1 dog", it's "1,000 dogs at once, or "1 dog at once, 1,000 different times". Rare but huge and coordinated siege, or a steady and predictable background radiation of small issues.

The latter is easier to handle, easier to fix, and much more suvivable if you do fuck it up a bit. It gives you some leeway to learn from mistakes.

If you make a mistake during the 1000 dog siege, or if you don't have enough guards on standby and ready to go just in case of this rare event, you're just cooked.

Re: Cloudflare outage on December 5, 2025

#433

Post by kvirani »

cpncrunch wrote: post #414
jacobgkau wrote: post #266

Earlier quoted context omitted.

I'm guessing you don't manage any production web servers? robots.txt isn't even respected by all of the American companies. Chinese ones (which often also use what are essentially botnets in Latin American and the rest of the world to evade detection) certainly don't care about anything short of dropping their packets.

I have been managing production commercial web servers for 28 years. Yes, there are various bots, and some of the large US companies such as Perplexity do indeed seem to be ignoring robots.txt. Is that a problem? It's certainly not a problem with cpu or network bandwidth (it's very minimal). Yes, it may be an issue if you are concerned with scraping (which I'm not). Cloudflare's "solution" is a much bigger problem th…

Security almost always brings inconvenience (to everyone involved, including end users). That is part of its cost.

Re: Cloudflare outage on December 5, 2025

#434

Post by lapcat »

JohnMakin wrote: post #417
lapcat wrote: post #200

Earlier quoted context omitted.

With all due respect, it sounds like you have never heard of unit tests. Cloudflare states that the compiler would prevent the bug in certain programming languages. So it seems ridiculous to suggest that the bug can't be detected outside the scale of a larger system.

Please explain how unit tests stop a problem from propagating across a system that fields 70 million requests a second and I’ll take you more seriously, otherwise I’m done with this particular subthread.

You're completely missing the point. From the blog post:

  if rule_result.action == "execute" then
    rule_result.execute.results = ruleset_results[tonumber(rule_result.execute.results_index)]
  end
"This code expects that, if the ruleset has action=”execute”, the “rule_result.execute” object will exist. However, because the rule had been skipped, the rule_result.execute object did not exist, and Lua returned an error due to attempting to look up a value in a nil value.

This is a straightforward error in the code, which had existed undetected for many years. This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur."

The unit tests are for the source code. In this respect, the number of requests a second fielded by the system is irrelevant. Unit tests don't happen in production; that's the point of them.

It's a classic coding mistake, failing to check for nil, and none of your handwaving about "scale" changes that fact.

Re: Cloudflare outage on December 5, 2025

#435

Post by cpncrunch »

filleduchaos wrote: post #428
cpncrunch wrote: post #414

Earlier quoted context omitted.

I have been managing production commercial web servers for 28 years. Yes, there are various bots, and some of the large US companies such as Perplexity do indeed seem to be ignoring robots.txt. Is that a problem? It's certainly not a problem with cpu or network bandwidth (it's very minimal). Yes, it may be an issue if you are concerned with scraping (which I'm not). Cloudflare's "solution" is a much bigger problem th…

It is rather disingenuous to backpedal from "you can easily block them" to "is that a problem? who even cares" when someone points out that you cannot in fact easily block them.

I was referring to legitimate ones, which you can easily block. Obviously there are scammy ones as well, and yes it is an issue, but for most sites I would say the cloudflare cure is worse than the problem it's trying to cure.

Re: Cloudflare outage on December 5, 2025

#436

Post by phatfish »

mewpmewp2 wrote: post #348
notepad0x90 wrote: post #264

Earlier quoted context omitted.

You really should take some of your pill. > Instead, it was triggered by changes being made to our body parsing logic while attempting to detect and mitigate an industry-wide vulnerability disclosed this week in React Server Components. > Unfortunately, in our FL1 version of our proxy, under certain circumstances, the second change of turning off our WAF rule testing tool caused an error state that resulted in 500 HT…

To be clear: 1. There is an active vulnerability unrelated to Cloudflare where React/Next.JS can be abused via a malicious payload. The payload could be up to 1MB. 2. Cloudflare had buffer size that wasn't enough to prevent that payload from being sent to the Customer of the Cloudflare. 3. Cloudflare to protect their customers wanted to increase the buffer size to 1MB. 4. Internal Testing Tool wasn't able to handle c…

Maybe they should have focused on fixing the "legacy tech debt mess", rather than pushing out more and more services and trying to be like AWS or Azure.

Re: Cloudflare outage on December 5, 2025

#437

Post by cpncrunch »

literalAardvark wrote: post #429
cpncrunch wrote: post #422

Earlier quoted context omitted.

There shouldn't be any noticeable hit on your cpu from bots from a site like that. Are you sure it's not a DDoS? Obviously it depends on the bot, and you can't block the scammy ones. I was really just referring to the major legitimate companies (which might not include Perplexity).

There is a noticeable hit, there's also a noticeable cost, and it's not a ddos. Not all sites can have full caching, we've tried.

I was referring to the community wiki.

Re: Cloudflare outage on December 5, 2025

#438

Post by winddude »

andrewf wrote: post #381
winddude wrote: post #322

Earlier quoted context omitted.

They don't, akamai has had several outages as well jsut no one notices. Akamai is way way smaller than cloudflare, 20% of internet traffic passes through CF networks, not sure it's even measurable on Akamai.

Quickly Googling about, a commonly repeated figure is that Akamai served 15% - 30% of Internet traffic in the late 2010's. They probably have less of the market today due to others growing, but they're not a minnow. 2024 revenue figures were $1.669 billion for Cloudflare, and $3.99 billion for Akamai, per Wikipedia.

https://w3techs.com/technologies/overview/proxy, they are tiny compared to CF, their revenue is high because they focus on large enterprise clients.

Re: Cloudflare outage on December 5, 2025

#439

Post by embedding-shape »

tobyjsullivan wrote: post #396
mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

I’m not sure I share this sentiment. First, let’s set aside the separate question of whether monopolies are bad. They are not good but that’s not the issue here. As to architecture: Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. But there’s a more interesting argume…

> Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so.

Why is that the only option? Cloudflare could offer solutions that let people run their software themselves, after paying some license fee. Or there could be many companies people use instead, instead of everyone flocking to one because of cargoculting "You need a CDN like Cloudflare before you launch your startup bro".

Re: Cloudflare outage on December 5, 2025

#440

Post by [deleted] »

cpncrunch wrote: post #422
mplewis wrote: post #359

Earlier quoted context omitted.

No you cannot! I blocked all of the user agents on a community wiki I run, and the traffic came back hours later masquerading as Firefox and Chrome. They just fucking lie to you and continue vacuuming your CPU.

There shouldn't be any noticeable hit on your cpu from bots from a site like that. Are you sure it's not a DDoS? Obviously it depends on the bot, and you can't block the scammy ones. I was really just referring to the major legitimate companies (which might not include Perplexity).

[deleted]
Post reply on HN