Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

361–370 of 601 posts

Re: Cloudflare outage on December 5, 2025

#361

Post by colechristensen »

mikkupikku wrote: post #330
Etheryte wrote: post #209

Earlier quoted context omitted.

This paraphrased urban legend has nothing to do with quality engineering though? As described, it's designed to the spec and working as intended.

It tracks with my experience in software quality engineering. Asked to find problems with something already working well in the field. Dutifully find bugs/etc. Get told that it's working though so nobody will change anything. In dysfunctional companies, which is probably most of them, quality engineering exists to cover asses, not to actually guide development.

It is not dysfunctional to ignore unreachable "bugs". A memory leak on a missile which won't be reached because it will explode long before that amount of time has passed is not a bug.

Re: Cloudflare outage on December 5, 2025

#362

Post by mewpmewp2 »

autoexec wrote: post #250
pulkitsh1234 wrote: post #229

Earlier quoted context omitted.

Genuinely curious, how to actually implement detection systems for a large scale global infra which that works with < 1 minute SLO ? Given cost is no constraint.

Right now I'd say maybe don't push changes to your entire global infra all at once and certainty not without testing your change first to make sure it doesn't break anything, but it's really not about a specific failure/fix as much as it is about a single company getting too big to do the job well or just plain doing more than it should in the first place. Honestly we shouldn't have created a system where any single…

But then would a customer have to use 10 different vendors to get the same things that Cloudflare currently provides? E.g. protection against various threats online?

Re: Cloudflare outage on December 5, 2025

#364

Post by chrneu »

cpncrunch wrote: post #211
james2doyle wrote: post #150

Earlier quoted context omitted.

You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? I would say Cloudflare is giving these site owners an option to protect their content and as a byproduct, reduce their own costs of subsidizing their thieves. They can choose to turn off the crawl protection. If they aren't, that tells you that they want it, doesn’t it?

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

this is the equivalent of asking people not to speed on your street.

Re: Cloudflare outage on December 5, 2025

#365

Post by colechristensen »

SketchySeaBeast wrote: post #252
wizzwizz4 wrote: post #249

Earlier quoted context omitted.

They will remember, because it'll have been measured and documented, rigorously.

I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.

>I wonder how that information is easily found afterwards.

Military hardware is produced with engineering design practices that look nothing at all like what most of the HN crowd is used to. There is an extraordinary amount of documentation, requirements, and validation done for everything.

There is a MIL-SPEC for pop tarts which defines all parts sizes, tolerances, etc.

Unlike a lot in the software world military hardware gets DONE with design and then they just manufacture it.

Re: Cloudflare outage on December 5, 2025

#366

Post by mixedbit »

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outages. Rust won't help, people will always make mistakes, also in Rust. Robust architecture addresses this by not allowing a single mistake to bring down myriad of unrelated services at once.

Re: Cloudflare outage on December 5, 2025

#367

Post by sznio »

beardedetim wrote: post #293
bombcar wrote: post #289

Earlier quoted context omitted.

They have millions of “free” subscribers; said subscribers should be the test pigs for rollouts; paying (read: big) subscribers can get the breaking changes later.

This feels like such a valid solution and is how past $dayjobs released things: send to the free users, rollout to Paying Users once that's proven to not blow up.

If your target is availability, that's correct.

If your target is security, then _assuming your patch is actually valid_ you're giving better security coverage for free customers than to your paying ones.

Cloudflare is both, and their tradeoffs seem to be set on maximizing security at cost of availability. And it makes sense. A fully unavailable system is perfectly secure.

Re: Cloudflare outage on December 5, 2025

#368

Post by mewpmewp2 »

udev4096 wrote: post #115
egorfine wrote: post #92

> provides customers with protection against malicious payloads, allowing them to be detected and blocked. To do this, Cloudflare’s proxy buffers HTTP request body content in memory for analysis. I have a mixed feeling about this. On the other hand, I absolutely don't want a CDN to look inside my payloads and decide what's good for me or. Today it's protection, tomorrow it's censorship. At the same time this is exact…

We need a decentralized ddos mitigation network based on incentives. Donate X amount of bandwidth, get Y amount of protection from other peers. Yes, we gotta do TLS inspection on every end for effective L7 mitigation but at least filtering can be done without decrypting any packets

How would that work for latency / reliability of the requests?

Re: Cloudflare outage on December 5, 2025

#369

Post by WD-42 »

mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

In other words, the consolidation on Cloudflare and AWS makes the web less stable. I agree.

Re: Cloudflare outage on December 5, 2025

#370

Post by wkat4242 »

colechristensen wrote: post #361
mikkupikku wrote: post #330

Earlier quoted context omitted.

It tracks with my experience in software quality engineering. Asked to find problems with something already working well in the field. Dutifully find bugs/etc. Get told that it's working though so nobody will change anything. In dysfunctional companies, which is probably most of them, quality engineering exists to cover asses, not to actually guide development.

It is not dysfunctional to ignore unreachable "bugs". A memory leak on a missile which won't be reached because it will explode long before that amount of time has passed is not a bug.

It's a debt though. Because people will forget it's there and then at some point someone changes a counter from milliseconds to microseconds and then the issue happens 1000 times sooner.

It's never right to leave structural issues even if "they don't happen under normal conditions".

Post reply on HN