Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

291–300 of 601 posts

Re: Cloudflare outage on December 5, 2025

#291

Post by wizzwizz4 »

SketchySeaBeast wrote: post #252
wizzwizz4 wrote: post #249

Earlier quoted context omitted.

They will remember, because it'll have been measured and documented, rigorously.

I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.

For the new system to be approved, you need to document the properties of the software component that are deemed relevant. The software system uses dynamic allocation, so "what do the allocation patterns look like? are there leaks, risks of fragmentation, etc, and how do we characterise those?" is on the checklist. The new developer could try to figure this all out from scratch, but if they're copying the old system's code, they're most likely just going to copy the existing paperwork, with a cursory check to verify that their modifications haven't changed the properties.

They're going to see "oh, it leaks 3MiB per minute… and this system runs for twice as long as the old system", and then they're going to think for five seconds, copy-paste the appropriate paragraph, double the memory requirements in the new system's paperwork, and call it a day.

Checklists work.

Re: Cloudflare outage on December 5, 2025

#292

Post by vimwizard »

switchbak wrote: post #284
lloeki wrote: post #260

Earlier quoted context omitted.

By reading the documentation thoroughly as a compulsory first step to designing the next system that depends on it. I realise this may probably boggle the mind of the modern software developer.

Just try harder. And if it still breaks, clearly you weren't trying hard enough! At some point you have to admit that humans are pretty bad at some things. Keeping documentation up to date and coherent is one of those things, especially in the age of TikTok. Better to live in the world we have and do the best you can, than to endlessly argue about how things should be but never will become.

> especially in the age of TikTok

Shouldn't grey beards, grizzled by years of practicing rigorous engineering, be passing this knowledge on to the next generation? How did they learn it when just starting out? They weren't born with it. Maybe engineering has actually improved so much that we only need to experience outages this frequently, and such feelings of nostalgia are born from never having to deal with systems having such high degrees of complexity and, realistically, 100% availability expectations on a global scale.

Re: Cloudflare outage on December 5, 2025

#293

Post by beardedetim »

bombcar wrote: post #289
uyzstvqs wrote: post #122

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately roll…

They have millions of “free” subscribers; said subscribers should be the test pigs for rollouts; paying (read: big) subscribers can get the breaking changes later.

This feels like such a valid solution and is how past $dayjobs released things: send to the free users, rollout to Paying Users once that's proven to not blow up.

Re: Cloudflare outage on December 5, 2025

#295

Post by XCSme »

bombcar wrote: post #287
XCSme wrote: post #254

Earlier quoted context omitted.

Plus, when you self-host, you can likely fix the issue yourself in a couple of hours max, instead of waiting indefinitely for a fix or support that might never come.

These global cloud outages aren’t the real issue; they affect everyone and get fixed. What is killer is when there is a KNOWN issue that affects YOU but basically only you so why bother fixing it!

I mean, I still prefer to have the ability to fix it myself, because I know I can probably do it in 1h max. I know this doesn't apply to most people, especially those outside of HN though.

Re: Cloudflare outage on December 5, 2025

#296

Post by [deleted] »

jacobgkau wrote: post #288

I noticed this outage last night (Cloudflare 500s on a few unrelated websites). As usual, when I went to Cloudflare's status page, nothing about the outage was present; the only thing there was a notice about the pre-planned maintenance work they were doing for the security issue, reporting that everything was being routed around it successfully.

[deleted]

Re: Cloudflare outage on December 5, 2025

#297

Post by venturecruelty »

system2 wrote: post #180
65 wrote: post #169

Earlier quoted context omitted.

My guess? Code written by AI

100%. Upper managements try to cut costs and hire remote bullshitters.

Agreed in re cost cutting, but there's no need to disparage those of us who don't want to be traffic for two hours every day.

Re: Cloudflare outage on December 5, 2025

#298

Post by venturecruelty »

samdoesnothing wrote: post #272
agentifysh wrote: post #232

Earlier quoted context omitted.

yet my dedicated server has been up since 2015 with zero downtimes i dont think this is an entropy issue its human error bubbling up and cloudflare charges a premium for it my faith in cloudflare is shoook for sure two major outages weeks apart ad this wont be the last

With all due respect, your dedicated server is not quite as complex as Cloudflare...

Eppur si muove. A random server serving things is exactly what the internet was supposed to be: a decentralized network of nodes.

Re: Cloudflare outage on December 5, 2025

#299

Post by venturecruelty »

hnthrowaway0328 wrote: post #140
liampulles wrote: post #90

The lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase. The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.

I'm not sure how decentralization helps though. People in a bazzar are going to care even less about sharing shadow knowledge. Linux IMO succeeds not because of the bazaar but because of Linus.

Decentralization is resilience; that's why the internet even works at all. That was the entire point of it, in fact.

Re: Cloudflare outage on December 5, 2025

#300

Post by spockz »

vimwizard wrote: post #292
switchbak wrote: post #284

Earlier quoted context omitted.

Just try harder. And if it still breaks, clearly you weren't trying hard enough! At some point you have to admit that humans are pretty bad at some things. Keeping documentation up to date and coherent is one of those things, especially in the age of TikTok. Better to live in the world we have and do the best you can, than to endlessly argue about how things should be but never will become.

> especially in the age of TikTok Shouldn't grey beards, grizzled by years of practicing rigorous engineering, be passing this knowledge on to the next generation? How did they learn it when just starting out? They weren't born with it. Maybe engineering has actually improved so much that we only need to experience outages this frequently, and such feelings of nostalgia are born from never having to deal with systems…

They may not have learned it but being thorough in general was more of a thing. These days things are far more rushed. And I say that as a relatively young engineer.

The amount of dedication and meticulous and concentrated work I know from older engineers when I started work and that I remember from my grand fathers is something I very rarely observe these days. Neither in engineering specific fields nor in general.

Post reply on HN