Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflare.com

Post reply on HN
Print view View original

481–490 of 560 posts

Re: Cloudflare was down

#481

Post by rvz »

cryptonym wrote: post #306
rvz wrote: post #126

Earlier quoted context omitted.

We are now seeing which companies do not consider the third party risk of single point of failures in systems they do not control as part of their infrastructure and what their contingency plan is. It turns out so far, there isn't one. Other than contacting the CEO of Cloudflare rather than switching on a temporary mitigation measure to ensure minimal downtime. Therefore, many engineers at affected companies would ha…

Sometimes it's not worth it. Your plan is just to accept you'll be off for a day or two, while you switch to a competitor.

Can't say that when it is a time critical service such as hospitals, banks, financial institutions or air-traffic control services.

Re: Cloudflare was down

#482

Post by JimDabell »

cryptonym wrote: post #464
meindnoch wrote: post #456

Earlier quoted context omitted.

React (a frontend JS framework) can now bring down critical Internet infrastructure. I will repeat it because it's so surreal: React (a frontend JS framework) can now bring down critical Internet infrastructure.

That's Next.js, not React. Mentioning React Server Components in the status page can be seen as a bad way to shift the blame. Would have been better to not specify which CVE they were trying to patch. The issue is their rollout management, not the Vendor and CVE.

> That's Next.js, not React.

React seems to think that it was React:

https://react.dev/blog/2025/12/03/critical-security-vulnerab...

Re: Cloudflare was down

#483

Post by ezst »

gbrindisi wrote: post #477
berkes wrote: post #175

Earlier quoted context omitted.

At least this warrants a good review of anyone's dependency on cloudflare. If it turns out that this was really just random bad luck, it shouldn't affect their reputation (if humans were rational, that is...) But if it is what many people seem to imply, that this is the outcome of internal problems/cuttings/restructuring/profit-increase etc, then I truly very much hope it affects their reputation. But I'm afraid it w…

The crowdstrike incident taught us that no one is going to review any dependency whatsoever.

Yep, that's what late stage capitalism leaves you with: consolidation, abuse, helplessness and complacency/widespread incompetence as a result

Re: Cloudflare was down

#485

Post by xomiachuna »

cherioo wrote: post #424
f311a wrote: post #421

> A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.

Where’s the source for this? It doesn’t look good when similar WAF issues caused their big outage a few years back.

https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

Re: Cloudflare was down

#486

Post by lima »

reassess_blind wrote: post #458
luastoned wrote: post #439

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com…

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up".

I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.

Re: Cloudflare was down

#488

Post by tialaramex »

Xunjin wrote: post #474
rhdunn wrote: post #471

Earlier quoted context omitted.

Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur. The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem the…

Agree 100%, however using your example, there is no regulatory agency that investigate the issue and demand changes to avoid related future problems. Should the industry move towards this way?

However, one of the things you see (if you read enough of them) in accident investigation reports for regulated industries is a recurring pattern

1. Accident happens 2. Investigators conclude Accident would not happen if people did X. Recommend regulator requires that people do X, citing previous such recommendations each iteration 3. Regulator declined this recommendation, arguing it's too expensive to do X, or people already do X, or even (hilariously) both 4. Go to 1.

Too often, what happens is that eventually

5. Extremely Famous Accident Happens, e.g. killing loved celebrity Space Cowboy 6. Investigators conclude Accident would not happen if people did X, remind regulator that they have previously recommended requiring X 7. Press finally reads dozens of previous reports and so News Story says: Regulator killed Space Cowboy! 8. Regulator decides actually they always meant to require X after all

Re: Cloudflare was down

#489

Post by rco8786 »

xomiachuna wrote: post #487

And so it seems that the cause is close to RSC vulnerability from yesterday: https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q So much for the react being just a frontend library, amirite

React server components initially came out 5ish years ago, for whatever that's worth.

Re: Cloudflare was down

#490

Post by stogot »

reassess_blind wrote: post #458
luastoned wrote: post #439

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com…

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

The update they describe should never bring down all services. I agree with other posters that they must lack a rollout strategy yet they sent spam emails mocking the reliability of other clouds
Post reply on HN