Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

31–40 of 601 posts

Re: Cloudflare outage on December 5, 2025

#31

Post by paradite »

The deployment pattern from Cloudflare looks insane to me.

I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes.

The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates.

I've never seen a downtime longer than 1 minute while I was there, because you get a spike on the dashboard immediately when something goes wrong.

For the entire system to be down for 10+ minutes due to a bad config change or deployment is just beyond me.

Re: Cloudflare outage on December 5, 2025

#32

Post by skywhopper »

xnorswap wrote: post #11

My understanding, paraphrased: "In order to gradually roll out one change, we had to globally push a different configuration change, which broke everything at once". But a more important takeaway: > This type of code error is prevented by languages with strong type systems

This is the exact same type of error that happened in their Rust code last time. Strong type systems don’t protect you from lazy programming.

Re: Cloudflare outage on December 5, 2025

#34

Post by skywhopper »

fidotron wrote: post #15

> This change was being rolled out using our gradual deployment system, and, as part of this rollout, we identified an increase in errors in one of our internal tools which we use to test and improve new WAF rules. As this was an internal tool, and the fix being rolled out was a security improvement, we decided to disable the tool for the time being as it was not required to serve or protect customer traffic. Come on…

China is probably a completely separate partition of their network.

Re: Cloudflare outage on December 5, 2025

#35

Post by denysvitali »

websiteapi wrote: post #20

i wonder why they cannot partially rollout. like the other outage they have to do a global rollout.

The article mentions that this Lua-based proxy is the old generation one, which is going to be replaced by the Rust based one (FL2) and that didn't fail on this scenario.

So, if anything, their efforts towards a typed language were justified. They just didn't manage to migrate everything in time before this incident - which is ironically a good thing since this incident was cause mostly by a rushed change in response to an actively exploited vulnerability.

Re: Cloudflare outage on December 5, 2025

#36

Post by fidotron »

skywhopper wrote: post #34
fidotron wrote: post #15

> This change was being rolled out using our gradual deployment system, and, as part of this rollout, we identified an increase in errors in one of our internal tools which we use to test and improve new WAF rules. As this was an internal tool, and the fix being rolled out was a security improvement, we decided to disable the tool for the time being as it was not required to serve or protect customer traffic. Come on…

China is probably a completely separate partition of their network.

One that doesn't get proactive security rollouts, it would seem.

Re: Cloudflare outage on December 5, 2025

#37

Post by samrus »

deadbabe wrote: post #18
flaminHotSpeedo wrote: post #13

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

As usual, Cloudflare is the man in the arena.

There are other men in the arena who arent tripping on their own feet

Re: Cloudflare outage on December 5, 2025

#38

Post by skywhopper »

dap wrote: post #14
jpeter wrote: post #2

Unwrap() strikes again

I guess you’re being facetious but for those who didn’t click through: > This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur.

That bit may be true, but the underlying error of a null reference that caused a panic was exactly the same in both incidents.

Re: Cloudflare outage on December 5, 2025

#39

Post by trashburger »

denysvitali wrote: post #24

Ironically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one). I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this. HugOps. Although bad for reputation, I think these incidents will help the…

I would very much like for him not to ignore the negativity, given that, you know, they are breaking the entire fucking Internet every time something like this happens.
Post reply on HN