Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

201–210 of 601 posts

Re: Cloudflare outage on December 5, 2025

#201

Post by marcosdumay »

newsoftheday wrote: post #175
liampulles wrote: post #77

Earlier quoted context omitted.

Rollback is a reliable strategy when the rollback process is well understood. If a rollback process is not well known and well experienced, then it is a risk in itself. I'm not sure of the nature of the rollback process in this case, but leaning on ill-founded assumptions is a bad practice. I do agree that a global rollout is a problem.

Rollback carries with it the contextual understanding of complete atomicity; otherwise it's slightly better than a yeet. It's similar to backups that are untested.

Complete atomicity carries with it the idea that the world is frozen, and any data only needs to change when you allow it to.

That's to say, it's an incredibly good idea when you can physically implement it. It's not something that everybody can do.

Re: Cloudflare outage on December 5, 2025

#202

Post by JeremyNT »

theideaofcoffee wrote: post #157
prdonahue wrote: post #89

Earlier quoted context omitted.

And you moved at a glacial pace compared to Cloudflare. There are tradeoffs.

Yes, of course, I want the organization that inserted itself into handling 20% of the world's internet traffic to move fast and break things. Like breaking the internet on a bi-weekly basis. Yep, great tradeoff there. Give me a break.

Lest we forget, they initially rose to prominence by being cheaper than the existing solutions, not better, and I suppose this is a tradeoff a lot of their customers are willing to make.

Re: Cloudflare outage on December 5, 2025

#204

Post by Hamuko »

jsnell wrote: post #21
xnorswap wrote: post #11

My understanding, paraphrased: "In order to gradually roll out one change, we had to globally push a different configuration change, which broke everything at once". But a more important takeaway: > This type of code error is prevented by languages with strong type systems

That's a bizarre takeaway for them to suggest, when they had exactly the same kind of bug with Rust like three weeks ago. (In both cases they had code implicitly expecting results to be available. When the results weren't available, they terminated processing of the request with an exception-like mechanism. And then they had the upstream services fail closed, despite the failing requests being to optional sidecars ra…

Yeah, my first thought was that had they used Rust, maybe we would've seen them point out a rule_result.unwrap() as the issue.

Re: Cloudflare outage on December 5, 2025

#205

Post by mopsi »

braiamp wrote: post #190
w10-1 wrote: post #166

Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineeri…

This is funny, considering that someone that worked on the defense industry (guide missile system) found a memory leak on one of their products, at that time. They told him that they knew about it, but that it's timed just right with the range of the system it would be used, so it doesn't matter.

... until the extended-range version is ordered and no one remembers to fix the leak. :]

Re: Cloudflare outage on December 5, 2025

#206

Post by linhns »

markus_zhang wrote: post #93
paradite wrote: post #31

The deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime lon…

My guess is that CF has so many external customers that they need to move fast and try not to break things. My hunch is that their culture always favors moving fast. As long as they are not breaking too many things, customers won't leave them.

I think everyone favors moving fast. We humans want to see results of our action early.

Re: Cloudflare outage on December 5, 2025

#207

Post by kccqzy »

nish__ wrote: post #119

Is it crazy to anyone else that they deploy every 5 minutes? And that it's not just config updates, but actual code changes with this "execute" action.

Config updates are not so clear cut from code changes.

Once I worked with a team in the anti-abuse space where the policy is that code deployments must happen over 5 days and config updates can take a few minutes. Then an engineer on the team argued that deploying new Python code doesn’t count as a code change because the CPython interpreter did not change; it didn’t even restart. And indeed given how dynamic Python is, it is totally possible to import new Python modules that did not exist when the interpreter process is launched.

Re: Cloudflare outage on December 5, 2025

#208

Post by [deleted] »

edoceo wrote: post #188
vlovich123 wrote: post #160

Earlier quoted context omitted.

> doesn't really justify the decision to use the same deployment system responsible for the 11/18 outage There’s no other deployment system available. There’s a single system for config deployment and it’s all that was available as they haven’t yet done the progressive roll out implementation yet.

Ok. Sure But shouldn't they have some beta/staging/test area they could deploy to, run tests for an hour then do the global blast?

[deleted]

Re: Cloudflare outage on December 5, 2025

#209

Post by Etheryte »

braiamp wrote: post #190
w10-1 wrote: post #166

Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineeri…

This is funny, considering that someone that worked on the defense industry (guide missile system) found a memory leak on one of their products, at that time. They told him that they knew about it, but that it's timed just right with the range of the system it would be used, so it doesn't matter.

This paraphrased urban legend has nothing to do with quality engineering though? As described, it's designed to the spec and working as intended.

Re: Cloudflare outage on December 5, 2025

#210

Post by programd »

liampulles wrote: post #77
flaminHotSpeedo wrote: post #13

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

Rollback is a reliable strategy when the rollback process is well understood. If a rollback process is not well known and well experienced, then it is a risk in itself. I'm not sure of the nature of the rollback process in this case, but leaning on ill-founded assumptions is a bad practice. I do agree that a global rollout is a problem.

Global rollout of security code on a timeframe of seconds is part of Cloudflare's value proposition.

In this case they got unlucky with an incident before they finished work on planned changes from the last incident.

Post reply on HN