Cloudflare outage on December 5, 2025 • Hacker News Board

Re: Cloudflare outage on December 5, 2025

#231

Post by locknitpicker » Fri, Dec 05, 2025, 7:13 PM UTC

vlovich123 wrote: post #160

flaminHotSpeedo wrote: post #100

Earlier quoted context omitted.

To clarify, I'm not trying to imply that I definitely wouldn't have made the same decision, or that cowboy decisions aren't ever the right call. However, this preliminary report doesn't really justify the decision to use the same deployment system responsible for the 11/18 outage. Deployment safety should have been the focus of this report, not the technical details. My question that I want answered isn't "are there…

> doesn't really justify the decision to use the same deployment system responsible for the 11/18 outage There’s no other deployment system available. There’s a single system for config deployment and it’s all that was available as they haven’t yet done the progressive roll out implementation yet.

> There’s no other deployment system available.

Hindsight is always 20/20, but I don't know how that sort of oversight could happen in an organization whose business model rides on reliability. Small shops understand the importance of safeguards such as progressive deployments or one-box-style deployments with a baking period, so why not the likes of Cloudflare? Don't they have anyone on their payroll who warns about the risks of global deployments without safeguards?

Re: Cloudflare outage on December 5, 2025

#232

Post by agentifysh » Fri, Dec 05, 2025, 7:14 PM UTC

liampulles wrote: post #90

The lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase. The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.

yet my dedicated server has been up since 2015 with zero downtimes

i dont think this is an entropy issue its human error bubbling up and cloudflare charges a premium for it

my faith in cloudflare is shoook for sure two major outages weeks apart ad this wont be the last

Re: Cloudflare outage on December 5, 2025

#233

Post by [deleted] » Fri, Dec 05, 2025, 7:14 PM UTC

w10-1 wrote: post #166

Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineeri…

[deleted]

Re: Cloudflare outage on December 5, 2025

#234

Post by roguecoder » Fri, Dec 05, 2025, 7:15 PM UTC

skywhopper wrote: post #38

dap wrote: post #14

Earlier quoted context omitted.

I guess you’re being facetious but for those who didn’t click through: > This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur.

That bit may be true, but the underlying error of a null reference that caused a panic was exactly the same in both incidents.

Yep: it is wild for them to claim that a strongly-typed language would have saved them when it didn't.

Relying on language features instead of writing code well will always eventually backfire.

Re: Cloudflare outage on December 5, 2025

#235

Post by james2doyle » Fri, Dec 05, 2025, 7:16 PM UTC

cpncrunch wrote: post #211

james2doyle wrote: post #150

Earlier quoted context omitted.

You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? I would say Cloudflare is giving these site owners an option to protect their content and as a byproduct, reduce their own costs of subsidizing their thieves. They can choose to turn off the crawl protection. If they aren't, that tells you that they want it, doesn’t it?

>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots

This is just using robots.txt and asking "pretty please, don’t scrape me".

Here is an article (from TODAY) about the case where Perplexity is being accused of ignoring robots.txt: https://www.theverge.com/news/839006/new-york-times-perplexi...

If you think a robots.txt is the answer to stopping the billion-dollar AI machine from scraping you, I don’t know what to say.

Re: Cloudflare outage on December 5, 2025

#236

Post by newsoftheday » Fri, Dec 05, 2025, 7:17 PM UTC

marcosdumay wrote: post #201

newsoftheday wrote: post #175

Earlier quoted context omitted.

Rollback carries with it the contextual understanding of complete atomicity; otherwise it's slightly better than a yeet. It's similar to backups that are untested.

Complete atomicity carries with it the idea that the world is frozen, and any data only needs to change when you allow it to. That's to say, it's an incredibly good idea when you can physically implement it. It's not something that everybody can do.

No, complete atomicity doesn't require a frozen state, it requires common sense and fail-proof, fool-proof guarantees derived from assurances gained from testing.

There is another name for rolling forward, it's called tripping up.

Re: Cloudflare outage on December 5, 2025

#237

Post by borplk » Fri, Dec 05, 2025, 7:17 PM UTC

Every time they screw up they write an elaborate postmortem and pat themselves on the back. Don't get me wrong, better have the postmortem than not. But at this point it seems like the only thing they are good at is writing incident postmortem blog posts.

Re: Cloudflare outage on December 5, 2025

#238

Post by roguecoder » Fri, Dec 05, 2025, 7:20 PM UTC

iLoveOncall wrote: post #103

denysvitali wrote: post #24

Ironically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one). I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this. HugOps. Although bad for reputation, I think these incidents will help the…

> I truly believe they're really going to make resilience their #1 priority now I hope that was their #1 priority from the very start given the services they sell... Anyway, people always tend to overthink about those black-swan events. Yes, 2 happened in a quick succession, but what is the average frequency overall? Insignificant.

This is Cloudflare. They've repeatedly broken DNS for years.

Looking across the errors, it points to some underlying practices: a lack of systems metaphors, modularity, testability, and an reliance on super-generic configuration instead of software with enforced semantics.

Re: Cloudflare outage on December 5, 2025

#239

Post by roguecoder » Fri, Dec 05, 2025, 7:22 PM UTC

fidotron wrote: post #36

skywhopper wrote: post #34

Earlier quoted context omitted.

China is probably a completely separate partition of their network.

One that doesn't get proactive security rollouts, it would seem.

The deploys are very unlikely to be managed from the same system.

Re: Cloudflare outage on December 5, 2025

#240

Post by roguecoder » Fri, Dec 05, 2025, 7:23 PM UTC

JohnMakin wrote: post #133

lapcat wrote: post #28

> This is a straightforward error in the code, which had existed undetected for many years. This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur. Cloudflare deployed code that was literally never tested, not even once, neither manually nor by unit test, otherwise the straightforward error wo…

Large scale infrastructure changes are often by nature completely untestable. The system is too large, there are too many moving parts to replicate with any kind of sane testing, so often, you do find out in prod, which is why robust and fast rollback procedures are usually desirable and implemented.

Akamai manages it.