Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

581–590 of 601 posts

Re: Cloudflare outage on December 5, 2025

#581

Post by theoldgreybeard »

mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

Notwithstanding that most people using Cloudflare aren't even benefiting from what it actually provides. They just use it...because reasons.

Re: Cloudflare outage on December 5, 2025

#582

Post by ifwinterco »

chippiewill wrote: post #564
ifwinterco wrote: post #495

Earlier quoted context omitted.

It's weird reading these reports because they don't seem to test anything at all (or at least there's very little mention of testing). Canary deployment, testing environments, unit tests, integration tests, anything really? It sounds like they test by merging directly to production but surely they don't

The problem is that Cloudflare do incremental rollouts and loads of testing for _code_. But they don't do the same thing for configuration - they globally push out changes because they want rapid response. It's still a bit silly though, their claimed reasoning probably doesn't really stack up for most of their config changes - I don't see it to be that likely that a 0.1->1->10->100 rollout over the period of 10 minut…

Yeah to me it doesn't make any sense - configuration changes are just as likely to break stuff (as they've discovered the hard way) and both of these issues could have been found in a testing environment before being deployed to production

Re: Cloudflare outage on December 5, 2025

#583

Post by cpncrunch »

oasisbob wrote: post #556
cpncrunch wrote: post #435

Earlier quoted context omitted.

I was referring to legitimate ones, which you can easily block. Obviously there are scammy ones as well, and yes it is an issue, but for most sites I would say the cloudflare cure is worse than the problem it's trying to cure.

No true scotsman needs Cloudflare, as any true scotsman can block AI bots themselves is not a strong argument.

But is there any actual evidence that any major AI bots are bypassing robots.txt? It looked as if Perplexity was doing this, but after looking into it further it seems that likely isn't the case. Quite often people believe single source news stories without doing any due diligence or fact checking.

Re: Cloudflare outage on December 5, 2025

#584

Post by cpncrunch »

chamomeal wrote: post #579
cpncrunch wrote: post #138

I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI exto…

Feel like that’s the fault of LLMs, not cloudflare

Looking into this more, it does indeed seem to be a cloudflare problem. It looks like cloudflare made a significant error in their bot fingerprinting, and Perplexity wasn't actually bypassing robots.txt.

https://www.perplexity.ai/hub/blog/agents-or-bots-making-sen...

To be honest I find cloudflare a much more scammy company than Perplexity. I had a DDoS attack a few years ago which originated from their network, and they had zero interest in it.

Re: Cloudflare outage on December 5, 2025

#585

Post by locknitpicker »

fallous wrote: post #559
locknitpicker wrote: post #514

Earlier quoted context omitted.

> "My architecture depends upon a single point of failure" is a great way to get laughed out of a design meeting. This is a simplistic opinion. Claiming services like Cloudflare are modeled as single points of failure is like complaining that your use of electricity to power servers is a single point of failure. Cloudflare sells a global network of highly reliable edge servers running services like caching, firewall,…

Your electricity to servers IS a single point of failure, if all you do is depend upon the power company to reliably feed power. There is a reason that co-location centers have UPS and generator backups for power. It may have been unthinkable to some casual observers that creating a giant single point of failure for the internet was a bad idea but it was entirely thinkable to others.

> Your electricity to servers IS a single point of failure, if all you do is depend upon the power company to reliably feed power.

I think you quite didn't got the point. The whole point is that putting together a system architecture that considers Cloudflare is a single point of failure is like designing a system architecture that considers a power supplier a single point of failure. Technically they can be considered that if you really really want to, but not only are things irredeemably broken when those failure modes are triggered but also they themselves are by far expected to be the most reliable components of your systems due to their design and SLAs that is pointless to waste time and resources mitigating such a scenario.

Re: Cloudflare outage on December 5, 2025

#586

Post by locknitpicker »

kortilla wrote: post #569
locknitpicker wrote: post #514

Earlier quoted context omitted.

> "My architecture depends upon a single point of failure" is a great way to get laughed out of a design meeting. This is a simplistic opinion. Claiming services like Cloudflare are modeled as single points of failure is like complaining that your use of electricity to power servers is a single point of failure. Cloudflare sells a global network of highly reliable edge servers running services like caching, firewall,…

You do know that data centers use backup generators because electricity is a single point of failure right? They even have multiple power supplies plugged into different circuits.

> You do know that data centers use backup generators because electricity is a single point of failure right?

How many times do you account for the existence of backup generators in a data center when you design, say, a SPA? How do you expect to be treated if you even suggest such a thing in a design requirements meeting? Do you understand what I am saying?

Re: Cloudflare outage on December 5, 2025

#587

Post by colechristensen »

wkat4242 wrote: post #370
colechristensen wrote: post #361

Earlier quoted context omitted.

It is not dysfunctional to ignore unreachable "bugs". A memory leak on a missile which won't be reached because it will explode long before that amount of time has passed is not a bug.

It's a debt though. Because people will forget it's there and then at some point someone changes a counter from milliseconds to microseconds and then the issue happens 1000 times sooner. It's never right to leave structural issues even if "they don't happen under normal conditions".

Again, when you're building a missile nobody should "forget" a detail.

You have very clearly in the specification, "this missile SHALL not have a run time before reboot of greater than 36 hours ref. donut_count.c:423 integer counter overflows"

Seriously, there's a military standard for pop tarts and they'd get rejected if they had out of spec amounts of frosting on top. It is not the software world you live in.

It's not that they don't ever make mistakes, just an extraordinary amount of effort is put into not making mistakes and oftentimes things are done "wrong" on purpose because of tradeoffs ordinary silicon valley software engineers have no context about.

Re: Cloudflare outage on December 5, 2025

#588

Post by cpncrunch »

bobbob1921 wrote: post #576
cpncrunch wrote: post #138

I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI exto…

Ive been seeing more of those prove your human pages as well, but I generally assume they are there to combat a DDOS or other type of attack (or maybe ai/bot). I remember how annoying it was combating DDOS attacks, or hacked sites before Cloudflare existed. I also remember how annoying capcha s were, everywhere. Cloudflare is not perfect but net, I think it’s been a great improvement.

Ovh provides ddos protection without that nonsense, for free. Aws does it too, for a fee.

Re: Cloudflare outage on December 5, 2025

#589

Post by steelblueskies »

mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

Reductionist, but it's a backup problem.

Data matters? Have multiple copies, not all in the same place.

This is really no different, yet we don't have those redundancies in play.

Host, and paths.

Every other take is ultimately just shuffling justification around the least bad for everyone lack of backups for cost saving.

Re: Cloudflare outage on December 5, 2025

#590

Post by literalAardvark »

cpncrunch wrote: post #583
oasisbob wrote: post #556

Earlier quoted context omitted.

No true scotsman needs Cloudflare, as any true scotsman can block AI bots themselves is not a strong argument.

But is there any actual evidence that any major AI bots are bypassing robots.txt? It looked as if Perplexity was doing this, but after looking into it further it seems that likely isn't the case. Quite often people believe single source news stories without doing any due diligence or fact checking.

I haven't been in the weeds in a few months, but last time I was there we did have a lot of traffic from bots that didn't care about robots. Bytedance is one that comes to mind.
Post reply on HN