Cloudflare outage on December 5, 2025 • Hacker News Board

Re: Cloudflare outage on December 5, 2025

#191

Post by agoodusername63 » Fri, Dec 05, 2025, 6:22 PM UTC

fidotron wrote: post #74

denysvitali wrote: post #69

Earlier quoted context omitted.

I have never seen an Ops team being rewarded for avoiding incidents (focusing in tech debt reduction), but instead they get the opposite - blamed when things go wrong. I think it's human nature (it's hard to realize something is going well until it breaks), but still has a very negative psychological effect. I can barely imagine the stress the team is going through right now.

> I have never seen an Ops team being rewarded for avoiding incidents That's why their salaries are so high.

news to me.

Re: Cloudflare outage on December 5, 2025

#192

Post by aeyes » Fri, Dec 05, 2025, 6:22 PM UTC

How hard can it be for a company with 1000 engineers to create a canary region before blasting their centralized changes out to everyone.

Every change is a deployment, even if its config. Treat it as such.

Also you should know that a strongly typed language won't save you from every type of problem. And especially not if you allow things like unwrap().

It is just mind boggling that they very obviously have completely untested code which proxies requests for all their customers. If you don't want to write the tests then at least fuzz it.

Re: Cloudflare outage on December 5, 2025

#193

Post by [deleted] » Fri, Dec 05, 2025, 6:22 PM UTC

denysvitali wrote: post #24

Ironically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one). I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this. HugOps. Although bad for reputation, I think these incidents will help the…

[deleted]

Re: Cloudflare outage on December 5, 2025

#194

Post by jimmydorry » Fri, Dec 05, 2025, 6:29 PM UTC

theideaofcoffee wrote: post #157

prdonahue wrote: post #89

Earlier quoted context omitted.

And you moved at a glacial pace compared to Cloudflare. There are tradeoffs.

Yes, of course, I want the organization that inserted itself into handling 20% of the world's internet traffic to move fast and break things. Like breaking the internet on a bi-weekly basis. Yep, great tradeoff there. Give me a break.

While you're taking your break, exploits gain traction in the wild and one of the value propositions for using a service provider like CloudFlare is catching and mitigating theses exploits as fast as possible. From the OP, this outage was in relation to handling a nasty RCE.

Re: Cloudflare outage on December 5, 2025

#195

Post by Uptrenda » Fri, Dec 05, 2025, 6:30 PM UTC

Can't believe one shitty website can take down most of the mainstream web.

Re: Cloudflare outage on December 5, 2025

#196

Post by JohnMakin » Fri, Dec 05, 2025, 6:30 PM UTC

lapcat wrote: post #149

JohnMakin wrote: post #133

Earlier quoted context omitted.

Large scale infrastructure changes are often by nature completely untestable. The system is too large, there are too many moving parts to replicate with any kind of sane testing, so often, you do find out in prod, which is why robust and fast rollback procedures are usually desirable and implemented.

> Large scale infrastructure changes are often by nature completely untestable. You're changing the subject here and shifting focus from the specific to the vague. The two postmortems after the recent major Cloudflare outages both listed straightforward errors in source code that could have been tested and detected. Theoretical outages could theoretically have other causes, but these two specific outages had specific…

I am not changing any subject. These are glue logic scripts connecting massive pieces of infra together, spanning what is likely several teams and orgs over the course of many years. It is impossible to blurt something out like "well, source code testing" for something like this, when the source code inputs are not possibly testable outside the scale of the larger system. They're often completely unknowable as well.

With all due respect, it sounds like you have not worked on these types of systems, but out of curiosity - what type of test do you think would have prevented this?

Re: Cloudflare outage on December 5, 2025

#197

Post by zwnow » Fri, Dec 05, 2025, 6:31 PM UTC

w10-1 wrote: post #166

Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineeri…

"Kudos"? This is like the South Park episode in which the oil company guy just excuses himself while the company just continues to fuck up over and over again. There's nothing to praise, this shouldn't happen twice in a month. Its inexcusable.

Re: Cloudflare outage on December 5, 2025

#198

Post by nova22033 » Fri, Dec 05, 2025, 6:40 PM UTC

paradite wrote: post #31

The deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime lon…

Speaking of fintech

https://www.henricodolfing.ch/case-study-4-the-440-million-s...

Re: Cloudflare outage on December 5, 2025

#199

Post by liampulles » Fri, Dec 05, 2025, 6:44 PM UTC

hnthrowaway0328 wrote: post #140

liampulles wrote: post #90

The lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase. The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.

I'm not sure how decentralization helps though. People in a bazzar are going to care even less about sharing shadow knowledge. Linux IMO succeeds not because of the bazaar but because of Linus.

What is the shadow knowledge in this case?

Re: Cloudflare outage on December 5, 2025

#200

Post by lapcat » Fri, Dec 05, 2025, 6:45 PM UTC

JohnMakin wrote: post #196

lapcat wrote: post #149

Earlier quoted context omitted.

> Large scale infrastructure changes are often by nature completely untestable. You're changing the subject here and shifting focus from the specific to the vague. The two postmortems after the recent major Cloudflare outages both listed straightforward errors in source code that could have been tested and detected. Theoretical outages could theoretically have other causes, but these two specific outages had specific…

I am not changing any subject. These are glue logic scripts connecting massive pieces of infra together, spanning what is likely several teams and orgs over the course of many years. It is impossible to blurt something out like "well, source code testing" for something like this, when the source code inputs are not possibly testable outside the scale of the larger system. They're often completely unknowable as well.…

With all due respect, it sounds like you have never heard of unit tests.

Cloudflare states that the compiler would prevent the bug in certain programming languages. So it seems ridiculous to suggest that the bug can't be detected outside the scale of a larger system.