Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

171–180 of 601 posts

Re: Cloudflare outage on December 5, 2025

#171

Post by vlovich123 »

paradite wrote: post #31

The deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime lon…

That is also true at Cloudflare for what it’s worth. However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release, especially since there’s a 5 min lag (if I recall correctly) in the monitoring dashboards to get all the telemetry from thousands of servers worldwide.

Comparing the difficulty of running the world’s internet traffic with hundreds of customer products with your fintech experience is like saying “I can lift 10 pounds. I don’t know why these guys are struggling to lift 500 pounds”.

Re: Cloudflare outage on December 5, 2025

#172

Post by [deleted] »

uyzstvqs wrote: post #122

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately roll…

[deleted]

Re: Cloudflare outage on December 5, 2025

#173

Post by crote »

flaminHotSpeedo wrote: post #109
lukeasrodgers wrote: post #59

Earlier quoted context omitted.

Roll back is not always the right answer. I can’t speak to its appropriateness in this particular situation of course, but sometimes “roll forward” is the better solution.

Like the other poster said, roll back should be the right answer the vast majority of the time. But it's also important to recognize that roll forward should be a replacement for the deployment you decided not to roll back, not a parallel deployment through another system. I won't say never, but a situation where the right answer to avoid a rollback (that it sounds like was technically fine to do, just undesirable fr…

Is a roll back even possible at Cloudflare's size?

With small deployments it usually isn't too difficult to re-deploy a previous commit. But once you get big enough you've got enough developers that half a dozen PRs will have been merged since the start of the incident and now. How viable is it to stop the world, undo everything, and start from scratch any time a deployment causes the tiniest issues?

Realistically the best you're going to get is merging a revert of the problematic changeset - but with the intervening merges that's still going to bring the system in a novel state. You're rolling forwards, not backwards.

Re: Cloudflare outage on December 5, 2025

#174

Post by LelouBil »

Bender wrote: post #125

Suggestion for Cloudflare: Create an early adopter option for free accounts. Benefit: Earliest uptake of new features and security patches. Drawback: Higher risk of outages. I think this should be possible since they already differentiate between free, pro and enterprise accounts. I do not know how the routing for that works but I bet they could do this. Think crowd-sourced beta testers. Also a perk for anything PCI…

I would for sure enable this, my personal server can handle being unreachable for a few hours in exchange for (potentially) interesting features.

Re: Cloudflare outage on December 5, 2025

#175

Post by newsoftheday »

liampulles wrote: post #77
flaminHotSpeedo wrote: post #13

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

Rollback is a reliable strategy when the rollback process is well understood. If a rollback process is not well known and well experienced, then it is a risk in itself. I'm not sure of the nature of the rollback process in this case, but leaning on ill-founded assumptions is a bad practice. I do agree that a global rollout is a problem.

Rollback carries with it the contextual understanding of complete atomicity; otherwise it's slightly better than a yeet. It's similar to backups that are untested.

Re: Cloudflare outage on December 5, 2025

#176

Post by newsoftheday »

crote wrote: post #173
flaminHotSpeedo wrote: post #109

Earlier quoted context omitted.

Like the other poster said, roll back should be the right answer the vast majority of the time. But it's also important to recognize that roll forward should be a replacement for the deployment you decided not to roll back, not a parallel deployment through another system. I won't say never, but a situation where the right answer to avoid a rollback (that it sounds like was technically fine to do, just undesirable fr…

Is a roll back even possible at Cloudflare's size? With small deployments it usually isn't too difficult to re-deploy a previous commit. But once you get big enough you've got enough developers that half a dozen PRs will have been merged since the start of the incident and now. How viable is it to stop the world, undo everything, and start from scratch any time a deployment causes the tiniest issues? Realistically th…

If companies like Cloudflare haven't figured out how to do reliable rollbacks, there seems little hope for any of us.

Re: Cloudflare outage on December 5, 2025

#177

Post by markus_zhang »

lionkor wrote: post #101

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

TBF, it depends on the number of outages locally. In my area it is one outage every thunderstorm/snow storm, so unfortunately the up time of my laptop, even with the help of a large, portable battery charging station (which can charge multiple laptops at the same time), is not optimistic.

I sometimes fancy that I could just take cash, go into the wood, build a small solar array, collect & cleanse river water, and buy a starlink console.

Re: Cloudflare outage on December 5, 2025

#179

Post by testplzignore »

Scaevolus wrote: post #6

> Disabling this was done using our global configuration system. This system does not use gradual rollouts but rather propagates changes within seconds to the entire network and is under review following the outage we recently experienced on November 18. > As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following LUA exception: They r…

> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible.

This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions.

Their timeline:

> 08:47: Configuration change deployed and propagated to the network

> 08:48: Change fully propagated

> 08:50: Automated alerts

> 09:11: Configuration change reverted and propagation start

> 09:12: Revert fully propagated, all traffic restored

2 minutes for their automated alerts to fire is terrible. For a system that is expected to have no downtime, they should have been alerted to the spike in 500 errors within seconds before the changes even fully propagated. Ideally the rollback would have been automated, but even if it is manual, the dude pressing the deploy button should have had realtime metrics on a second display with his finger hovering over the rollback button.

Ok, so they want to take the approach of roll forward instead of immediate rollback. Again, that's a valid approach, but you need to be prepared. At 08:48, they would have had tens of millions of "init.lua:314: attempt to index field 'execute'" messages being logged per second. Exact line of code. Not a complex issue. They should have had engineers reading that code and piecing this together by 08:49. The change you just deployed was to disable an "execute" rule. Put two and two together. Initiate rollback by 08:50.

How disconnected are the teams that do deployments vs the teams that understand the code? How many minutes were they scratching their butts wondering "what is init.lua"? Are they deploying while their best engineers are sleeping?

Re: Cloudflare outage on December 5, 2025

#180

Post by system2 »

65 wrote: post #169
miyuru wrote: post #16

Whats going on with cloudflare's software team? I have seen similar bugs in cloudflare API recently as well. There is an endpoint for a feature that is available only to enterprise users, but the check for whether the user is on an enterprise plan is done at the last step.

My guess? Code written by AI

100%. Upper managements try to cut costs and hire remote bullshitters.
Post reply on HN