Cloudflare outage on December 5, 2025

Post by ignoramous » Fri, Dec 05, 2025, 5:29 PM UTC

flaminHotSpeedo wrote: post #13

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

  if rule_result.action == "execute" then
    rule_result.execute.results = ruleset_results[tonumber(rule_result.execute.results_index)]
  end

  This code expects that, if the ruleset has action="execute", the "rule_result.execute" object will exist ... error in the [Lua] code, which had existed undetected for many years ... prevented by languages with strong type systems. In our replacement [FL2 proxy] ... code written in Rust ... the error did not occur.

Post by NicoJuicy » Fri, Dec 05, 2025, 5:33 PM UTC

flaminHotSpeedo wrote: post #13

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

Post by drysart » Fri, Dec 05, 2025, 5:38 PM UTC

Pharaoh2 wrote: post #152

flaminHotSpeedo wrote: post #113

Earlier quoted context omitted.

Do you have a public source about an embargo period for this one? I wasn't able to find one

https://react.dev/blog/2025/12/03/critical-security-vulnerab... Privately Disclosed: Nov 29 Fix pushed: Dec 1 Publicly disclosed: Dec 3

Post by theideaofcoffee » Fri, Dec 05, 2025, 5:40 PM UTC

prdonahue wrote: post #89

theideaofcoffee wrote: post #81

Earlier quoted context omitted.

Same, my time at a F100 ecommerce retailer showed me the same. Every change control board justification needed an explicit back-out/restoration plan with exact steps to be taken, what was being monitored to ensure that was being held to, contacts of prominent groups anticipated to have an effect, emergency numbers/rooms for quick conferences if in fact something did happen. The process was pretty tight, almost no rev…

And you moved at a glacial pace compared to Cloudflare. There are tradeoffs.

Post by pdimitar » Fri, Dec 05, 2025, 5:42 PM UTC

jsnell wrote: post #21

xnorswap wrote: post #11

My understanding, paraphrased: "In order to gradually roll out one change, we had to globally push a different configuration change, which broke everything at once". But a more important takeaway: > This type of code error is prevented by languages with strong type systems

That's a bizarre takeaway for them to suggest, when they had exactly the same kind of bug with Rust like three weeks ago. (In both cases they had code implicitly expecting results to be available. When the results weren't available, they terminated processing of the request with an exception-like mechanism. And then they had the upstream services fail closed, despite the failing requests being to optional sidecars ra…