Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

81–90 of 601 posts

Re: Cloudflare outage on December 5, 2025

#81

Post by theideaofcoffee »

paradite wrote: post #31

The deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime lon…

Same, my time at a F100 ecommerce retailer showed me the same. Every change control board justification needed an explicit back-out/restoration plan with exact steps to be taken, what was being monitored to ensure that was being held to, contacts of prominent groups anticipated to have an effect, emergency numbers/rooms for quick conferences if in fact something did happen.

The process was pretty tight, almost no revenue-affecting outages from what I can remember because it was such a collaborative effort (even though the board presentation seemed a bit spiky and confrontational at the time, everyone was working together).

Re: Cloudflare outage on December 5, 2025

#84

Post by beanjuiceII »

denysvitali wrote: post #47
trashburger wrote: post #39

Earlier quoted context omitted.

I would very much like for him not to ignore the negativity, given that, you know, they are breaking the entire fucking Internet every time something like this happens.

This is the kind of comment I wish he would ignore. You can be angry - but that doesn't help anyone. They fucked up, yes, they admitted it and they provided plans on how to address that. I don't think they do these things on purpose. Of course given their good market penetration they end up disrupting a lot of customers - and they should focus on slow rollouts - but I also believe that in a DDOS protection system (or…

I hope he doesn't ignore it, the internet has been forgiving enough toward cloudflares string of failures..its getting pretty old, and creates a ton of choas. I work with life saving devices, being impacted in any way in data monitoring has a huge impact in many ways. "sorry ma'am we can't give your child t1d readings on your follow app because our provider decided to break everything in the pursuit of some react bug." has a great ring to it

Re: Cloudflare outage on December 5, 2025

#85

Post by fidotron »

denysvitali wrote: post #79
fidotron wrote: post #74

Earlier quoted context omitted.

> I have never seen an Ops team being rewarded for avoiding incidents That's why their salaries are so high.

Depending on the tech debt, the ops team might just be in "survival mode" and not have the time to fix every single issue. In this particular case, they seem to be doing two things: - Phasing out the old proxy (Lua based) which is replaced by FL2 (Rust based, the one that caused the previous incident) - Reacting to an actively exploited vulnerability in React by deploying WAF rules - and they're doing them in a relat…

They deliberately ignored an internal tool that started erroring out at the given deployment and rolled it out anyway without further investigation.

That's not deserving of sympathy.

Re: Cloudflare outage on December 5, 2025

#88

Post by NoSalt »

flaminHotSpeedo wrote: post #13

What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…

Ooh ... I want to be on a cowboy decision making team!!!

Re: Cloudflare outage on December 5, 2025

#89

Post by prdonahue »

theideaofcoffee wrote: post #81
paradite wrote: post #31

The deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime lon…

Same, my time at a F100 ecommerce retailer showed me the same. Every change control board justification needed an explicit back-out/restoration plan with exact steps to be taken, what was being monitored to ensure that was being held to, contacts of prominent groups anticipated to have an effect, emergency numbers/rooms for quick conferences if in fact something did happen. The process was pretty tight, almost no rev…

And you moved at a glacial pace compared to Cloudflare. There are tradeoffs.

Re: Cloudflare outage on December 5, 2025

#90

Post by liampulles »

The lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase.

The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.

Post reply on HN