Cloudflare outage on December 5, 2025 • Hacker News Board

Re: Cloudflare outage on December 5, 2025

#51

Post by nine_k » Fri, Dec 05, 2025, 4:06 PM UTC

flaminHotSpeedo wrote: post #13

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

> more to the story

From a more tinfoil-wearing angle, it may not even be a regular deployment, given the idea of Cloudflare being "the largest MitM attack in history". ("Maybe not even by Cloudflare but by NSA", would say some conspiracy theorists, which is, of course, completely bonkers: NSA is supposed to employ engineers who never let such blunders blow their cover.)

Re: Cloudflare outage on December 5, 2025

#52

Post by rany_ » Fri, Dec 05, 2025, 4:07 PM UTC

> As part of our ongoing work to protect customers using React against a critical vulnerability, CVE-2025-55182, we started rolling out an increase to our buffer size to 1MB, the default limit allowed by Next.js applications.

Why would increasing the buffer size help with that security vulnerability? Is it just a performance optimization?

Re: Cloudflare outage on December 5, 2025

#53

Post by _pdp_ » Fri, Dec 05, 2025, 4:07 PM UTC

So no static compiler checks and apparently no fuzzers used to ensure these rules work as intended?

Re: Cloudflare outage on December 5, 2025

#54

Post by da_grift_shift » Fri, Dec 05, 2025, 4:07 PM UTC

denysvitali wrote: post #24

Ironically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one). I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this. HugOps. Although bad for reputation, I think these incidents will help the…

[ Removed by Reddit ]

Re: Cloudflare outage on December 5, 2025

#55

Post by chatmasta » Fri, Dec 05, 2025, 4:07 PM UTC

hrimfaxi wrote: post #33

Having their changes fully propagate within 1 minute is pretty fantastic.

The coolest part of Cloudflare’s architecture is that every server is the same… which presumably makes deployment a straightforward task.

Re: Cloudflare outage on December 5, 2025

#56

Post by philipwhiuk » Fri, Dec 05, 2025, 4:08 PM UTC

Scaevolus wrote: post #6

> Disabling this was done using our global configuration system. This system does not use gradual rollouts but rather propagates changes within seconds to the entire network and is under review following the outage we recently experienced on November 18. > As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following LUA exception: They r…

> Warning signs like this are how you know that something might be wrong!

Yes, as they explain it's the rollback that was triggered due to seeing these errors that broke stuff.

Re: Cloudflare outage on December 5, 2025

#57

Post by ramon156 » Fri, Dec 05, 2025, 4:08 PM UTC

antiloper wrote: post #19

Make faster websites: > we started rolling out an increase to our buffer size to 1MB, the default limit allowed by Next.js applications. Why is the Next.js limit 1 MB? It's not enough for uploading user generated content (photographs, scanned invoices), but a 1 MB request body for even multiple JSON API calls is ridiculous. There frameworks need to at least provide some pushback to unoptimized development, even if it…

The update was to update it to 3MB (paid 10MB)

Re: Cloudflare outage on December 5, 2025

#58

Post by skywhopper » Fri, Dec 05, 2025, 4:09 PM UTC

fidotron wrote: post #36

skywhopper wrote: post #34

Earlier quoted context omitted.

China is probably a completely separate partition of their network.

One that doesn't get proactive security rollouts, it would seem.

I assume it was next on the checklist, or assigned to a different ops team.

Re: Cloudflare outage on December 5, 2025

#59

Post by lukeasrodgers » Fri, Dec 05, 2025, 4:09 PM UTC

flaminHotSpeedo wrote: post #13

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

Roll back is not always the right answer. I can’t speak to its appropriateness in this particular situation of course, but sometimes “roll forward” is the better solution.

Re: Cloudflare outage on December 5, 2025

#60

Post by websiteapi » Fri, Dec 05, 2025, 4:09 PM UTC

denysvitali wrote: post #35

websiteapi wrote: post #20

i wonder why they cannot partially rollout. like the other outage they have to do a global rollout.

The article mentions that this Lua-based proxy is the old generation one, which is going to be replaced by the Rust based one (FL2) and that didn't fail on this scenario. So, if anything, their efforts towards a typed language were justified. They just didn't manage to migrate everything in time before this incident - which is ironically a good thing since this incident was cause mostly by a rushed change in response…

yes, but as the article states why are they doing global fast rollouts?