Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflare.com

Post reply on HN
Print view View original

501–510 of 560 posts

Re: Cloudflare was down

#501

Post by cryptonym »

JimDabell wrote: post #482
cryptonym wrote: post #464

Earlier quoted context omitted.

That's Next.js, not React. Mentioning React Server Components in the status page can be seen as a bad way to shift the blame. Would have been better to not specify which CVE they were trying to patch. The issue is their rollout management, not the Vendor and CVE.

> That's Next.js, not React. React seems to think that it was React: https://react.dev/blog/2025/12/03/critical-security-vulnerab...

True, thanks for sharing. Worth mentioning that's on the "full-stack" part of the framework. It doesn't impact most React website while it impacts most next.js websites.

Re: Cloudflare was down

#502

Post by lima »

kypro wrote: post #491
tialaramex wrote: post #459

Earlier quoted context omitted.

Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up. They have blameless post mortems, but maybe "We actually do m…

> They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear. Or they could say, "we want to continue to prioritise speed of security rollouts over stability, and despite our best efforts, we do make mistakes, so sometimes we expect things will blow up". I guess it depends what you're optimising for... If the rollout speed of security…

It's just poor risk management at this point. Making sure that a configuration change doesn't crash the production service shouldn't take more than a few seconds in a well-engineered system even if you're not doing staged rollout.

Re: Cloudflare was down

#504

Post by brandensilva »

stogot wrote: post #490
reassess_blind wrote: post #458

Earlier quoted context omitted.

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

The update they describe should never bring down all services. I agree with other posters that they must lack a rollout strategy yet they sent spam emails mocking the reliability of other clouds

The irony is they support rolling out incrementally with some of their products for deployment.

They need that same mindset for themselves in config/updates/infra changes but probably easier said than done.

Re: Cloudflare was down

#505

Post by jq-r »

meindnoch wrote: post #495
rhdunn wrote: post #471

Earlier quoted context omitted.

Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur. The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem the…

>don't blame the people involved (unless they are acting maliciously) Or negligently.

That still shouldn't be a part of post mortem, more of a performance review item.

Re: Cloudflare was down

#506

Post by brandensilva »

bluerooibos wrote: post #472
pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

I'm quite sure the reputational damage has already been done. How do they not have better isolation of these issues, or redundancy of some sort?

The seed has been planted. It will take awhile for others to fill the void. Still the big players see this as an opportunity to steal market share if Cloudflare cannot live up to their reputation.

Re: Cloudflare was down

#507

Post by gpi »

reassess_blind wrote: post #458
luastoned wrote: post #439

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com…

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

I believe they use Argo according to a previous post mortem.

https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-...

Re: Cloudflare was down

#508

Post by tempaccount420 »

jq-r wrote: post #505
meindnoch wrote: post #495

Earlier quoted context omitted.

>don't blame the people involved (unless they are acting maliciously) Or negligently.

That still shouldn't be a part of post mortem, more of a performance review item.

They should be performantly removed.

Re: Cloudflare was down

#509

Post by tempaccount420 »

cryptonym wrote: post #501
JimDabell wrote: post #482

Earlier quoted context omitted.

> That's Next.js, not React. React seems to think that it was React: https://react.dev/blog/2025/12/03/critical-security-vulnerab...

True, thanks for sharing. Worth mentioning that's on the "full-stack" part of the framework. It doesn't impact most React website while it impacts most next.js websites.

It was React. Code in React's repository had to be patched to fix this.

Next.JS just happens to be the biggest user of this part of React, but blaming Next.JS is weird...

Re: Cloudflare was down

#510

Post by ethbr1 »

lima wrote: post #486
reassess_blind wrote: post #458

Earlier quoted context omitted.

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do th…

For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout?

Blue/green and temporarily ossify capacity? Regional?

Post reply on HN