Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.
Cloudflare outage on December 5, 2025
101–110 of 601 posts
Re: Cloudflare outage on December 5, 2025
#102Is there some underlying factors that resulted in the recent outages (e.g., new processes, layoffs, etc.) or just a series of pure coincidences?
Re: Cloudflare outage on December 5, 2025
#103Ironically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one). I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this. HugOps. Although bad for reputation, I think these incidents will help the…
> I truly believe they're really going to make resilience their #1 priority now
I hope that was their #1 priority from the very start given the services they sell...
Anyway, people always tend to overthink about those black-swan events. Yes, 2 happened in a quick succession, but what is the average frequency overall? Insignificant.
Re: Cloudflare outage on December 5, 2025
#104> Disabling this was done using our global configuration system. This system does not use gradual rollouts but rather propagates changes within seconds to the entire network and is under review following the outage we recently experienced on November 18. > As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following LUA exception: They r…
> Warning signs like this are how you know that something might be wrong! Yes, as they explain it's the rollback that was triggered due to seeing these errors that broke stuff.
They saw errors and decided to do a second rollout to disable the component generating errors, causing a major outage.
Re: Cloudflare outage on December 5, 2025
#105> Disabling this was done using our global configuration system. This system does not use gradual rollouts but rather propagates changes within seconds to the entire network and is under review following the outage we recently experienced on November 18. > As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following LUA exception: They r…
> Warning signs like this are how you know that something might be wrong! Yes, as they explain it's the rollback that was triggered due to seeing these errors that broke stuff.
Would be nice if the outage dashboards are directly linked to this instead of whatever they have now.
Re: Cloudflare outage on December 5, 2025
#106Ironically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one). I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this. HugOps. Although bad for reputation, I think these incidents will help the…
> HugOps This childish nonsense needs to end. Ops are heavily rewarded because they're supposed to be responsible. If they're not then the associated rewards for it need to stop as well.
Ops has never been "rewarded" at any org I've ever been at or heard about, including physical infra companies.
Re: Cloudflare outage on December 5, 2025
#107What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…
One thing to keep in mind when judging what's 'appropriate' is that Cloudflare was effectively responding to an ongoing security incident outside of their control (the React Server RCE vulnerability). Part of Cloudlfare's value proposition is being quick to react to such threats. That changes the equation a bit: any hour you wait longer to deploy, your customers are actively getting hacked through a known high-severi…
[flagged]
Re: Cloudflare outage on December 5, 2025
#108Earlier quoted context omitted.
I have never seen an Ops team being rewarded for avoiding incidents (focusing in tech debt reduction), but instead they get the opposite - blamed when things go wrong. I think it's human nature (it's hard to realize something is going well until it breaks), but still has a very negative psychological effect. I can barely imagine the stress the team is going through right now.
> I have never seen an Ops team being rewarded for avoiding incidents That's why their salaries are so high.
Ops salaries are high??? Where?!?!
Re: Cloudflare outage on December 5, 2025
#109What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…
Roll back is not always the right answer. I can’t speak to its appropriateness in this particular situation of course, but sometimes “roll forward” is the better solution.
Like the other poster said, roll back should be the right answer the vast majority of the time. But it's also important to recognize that roll forward should be a replacement for the deployment you decided not to roll back, not a parallel deployment through another system.
I won't say never, but a situation where the right answer to avoid a rollback (that it sounds like was technically fine to do, just undesirable from a security/business perspective) is a parallel deployment through a radioactive, global blast radius, near instantaneous deployment system that is under intense scrutiny after another recent outage should be about as probable as a bowl of petunias in orbit
Re: Cloudflare outage on December 5, 2025
#110Earlier quoted context omitted.
Like who? Which large tech company doesn't have outages?
It's not about outages. It's about the why. Hardware can fail. Bugs can happen. But to continue a roll out despite warning sings and without understanding the cause and impact is on another level. Especially if it is related to the same problem as last time.
[dead]