Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

271–280 of 601 posts

Re: Cloudflare outage on December 5, 2025

#271

Post by therein »

morpheos137 wrote: post #268
testplzignore wrote: post #179

Earlier quoted context omitted.

> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions. Their timeline: > 08:47: Configuration change deplo…

I see lots of people complaining about this down time but in actuality is it really that big a deal to have 30 minutes of down time or whatever. It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake. In many developed countries the electric power service has local down times on occasion. That's more important than not being able…

> about this down time but in actuality is it really that big a deal to have 30 minutes of down time or whatever. It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake.

This reads like sarcasm. But I guess it is not. Yes, you are a CDN, a major one at that. 30 minutes of downtime or "whatever" is not acceptable. I worked at traffic teams of social networks that looked at themselves as that mission critical. CF is absolutely that critical and it is definitely lives at stake.

Re: Cloudflare outage on December 5, 2025

#272

Post by samdoesnothing »

agentifysh wrote: post #232
liampulles wrote: post #90

The lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase. The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.

yet my dedicated server has been up since 2015 with zero downtimes i dont think this is an entropy issue its human error bubbling up and cloudflare charges a premium for it my faith in cloudflare is shoook for sure two major outages weeks apart ad this wont be the last

With all due respect, your dedicated server is not quite as complex as Cloudflare...

Re: Cloudflare outage on December 5, 2025

#273

Post by echelon »

drysart wrote: post #158
echelon wrote: post #97

Earlier quoted context omitted.

You want to build a world where roll back is 95% the right thing to do. So that it almost always works and you don't even have to think about it. During an incident, the incident lead should be able to say to your team's on call: "can you roll back? If so, roll back" and the oncall engineer should know if it's okay. By default it should be if you're writing code mindfully. Certain well-understood migrations are the o…

"Fail open" state would have been improper here, as the system being impacted was a security-critical system: firewall rules. It is absolutely the wrong approach to "fail open" when you can't run security-critical operations.

Cloudflare is supposed to protect me from occasional ddos, not take my business offline entirely.

This can be architected in such a way that if one rules engine crashes, other systems are not impacted and other rules, cached rules, heuristics, global policies, etc. continue to function and provide shielding.

You can't ask for Cloudflare to turn on a dime and implement this in this manner. Their infra is probably very sensibly architected by great engineers. But there are always holes, especially when moving fast, migrating systems, etc. And there's probably room for more resiliency.

Re: Cloudflare outage on December 5, 2025

#274

Post by lukan »

lloeki wrote: post #260
SketchySeaBeast wrote: post #252

Earlier quoted context omitted.

I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.

By reading the documentation thoroughly as a compulsory first step to designing the next system that depends on it. I realise this may probably boggle the mind of the modern software developer.

You say this like trivial misstakes did not happen all the time in classical engineering as well.

If there is a memory leak, them this is a flaw, that might not matter so much for a specific product, but I can also easily see it being forgotten, if it was maybe mentioned somewhere in the documentation, but maybe not clear enough and deadlines and stress to ship are a thing there as well.

Re: Cloudflare outage on December 5, 2025

#275

Post by dingnuts »

jacobgkau wrote: post #266
cpncrunch wrote: post #211

Earlier quoted context omitted.

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

I'm guessing you don't manage any production web servers? robots.txt isn't even respected by all of the American companies. Chinese ones (which often also use what are essentially botnets in Latin American and the rest of the world to evade detection) certainly don't care about anything short of dropping their packets.

[dead]

Re: Cloudflare outage on December 5, 2025

#276

Post by hinkley »

mopsi wrote: post #205
braiamp wrote: post #190

Earlier quoted context omitted.

This is funny, considering that someone that worked on the defense industry (guide missile system) found a memory leak on one of their products, at that time. They told him that they knew about it, but that it's timed just right with the range of the system it would be used, so it doesn't matter.

... until the extended-range version is ordered and no one remembers to fix the leak. :]

Ariane 5 happens.

Re: Cloudflare outage on December 5, 2025

#277

Post by hinkley »

wizzwizz4 wrote: post #249
mopsi wrote: post #205

Earlier quoted context omitted.

... until the extended-range version is ordered and no one remembers to fix the leak. :]

They will remember, because it'll have been measured and documented, rigorously.

Was this one measured and documented rigorously?

Well obviously not, because the front fell off. That’s a dead giveaway.

Re: Cloudflare outage on December 5, 2025

#278

Post by hinkley »

SketchySeaBeast wrote: post #252
wizzwizz4 wrote: post #249

Earlier quoted context omitted.

They will remember, because it'll have been measured and documented, rigorously.

I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.

If ownerless code doesn’t result in discoverability efforts then the whole thing goes off the rails.

I won’t remember this block of code because five other people have touched it. So I need to be able to see what has changed and what it talks to so I can quickly verify if my old assumptions still hold true

Re: Cloudflare outage on December 5, 2025

#279

Post by hinkley »

lloeki wrote: post #260
SketchySeaBeast wrote: post #252

Earlier quoted context omitted.

I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.

By reading the documentation thoroughly as a compulsory first step to designing the next system that depends on it. I realise this may probably boggle the mind of the modern software developer.

That is not how this usually works.

Most of the time QA can tell you exactly how the product works, regardless of what the documentation says. But many of us haven’t seen a QA team in five, ten years.

Re: Cloudflare outage on December 5, 2025

#280

Post by pdimitar »

zozbot234 wrote: post #261
pdimitar wrote: post #159

Earlier quoted context omitted.

To be precise, the previous problem with Rust was because somebody copped out and used a temporary escape hatch function that absolutely has no place in production code. It was mostly an amateur mistake. Not Rust's fault. Rust could never gain adoption if it didn't have a few escape hatches. "Damned if they do, damned if they don't" kind of situation. There are even lints for the usage of the `unwrap` and `expect` fu…

You can make an argument that .unwrap() should have no place in production code, but .expect("invariant violated: etc. etc.") very much has its place. When the system is in an unpredicted and not-designed-for state it is supposed to shut down promptly, because this makes it easier to troubleshoot the root cause failure whereas not doing so may have even worse consequences.

I don't disagree but you might as well also manually send an error to f.ex. Sentry and just halt processing of the request.

Though that really depends. In companies where k8s is used the app will be brought back up immediately anyway.

Post reply on HN