Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflare.com

Post reply on HN
Print view View original

511–520 of 560 posts

Re: Cloudflare was down

#511

Post by JB_Dev »

lima wrote: post #486
reassess_blind wrote: post #458

Earlier quoted context omitted.

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do th…

Yea agree.. This is the same discussion point that came up last time they had an incident.

I really don’t buy this requirement to always deploy state changes 100% globally immediately. Why can’t they just roll out to 1%, scaling to 100% over 5 minutes (configurable), with automated health checks and pauses? That will go along way towards reducing the impact of these regressions.

Then if they really think something is so critical that it goes everywhere immediately, then sure set the rollout to start at 100%.

Point is, design the rollout system to give you that flexibility. Routine/non-critical state changes should go through slower ramping rollouts.

Re: Cloudflare was down

#512

Post by ethbr1 »

tialaramex wrote: post #488
Xunjin wrote: post #474

Earlier quoted context omitted.

Agree 100%, however using your example, there is no regulatory agency that investigate the issue and demand changes to avoid related future problems. Should the industry move towards this way?

However, one of the things you see (if you read enough of them) in accident investigation reports for regulated industries is a recurring pattern 1. Accident happens 2. Investigators conclude Accident would not happen if people did X. Recommend regulator requires that people do X, citing previous such recommendations each iteration 3. Regulator declined this recommendation, arguing it's too expensive to do X, or peop…

As bad as (3) sounds, I'll strongman the argument: it's important to keep the economic cost of any regulation in mind.*

On the one hand, you'd like to prevent the thing the regulation is seeking to prevent.

On the other hand, you'd have costs for the regulation to be implemented (one-time and/or ongoing).

"Is the good worth the costs?" is a question worth asking every time. (Not least because sometimes it lets you downscope/target regulations to get better good ROI)

*Yes, the easy pessimistic take is 'industry fights all regulation on cost grounds', but the fact that the argument is abused doesn't mean it doesn't have some underlying merit

Re: Cloudflare was down

#513

Post by creamyhorror »

cryptonym wrote: post #306
rvz wrote: post #126

Earlier quoted context omitted.

We are now seeing which companies do not consider the third party risk of single point of failures in systems they do not control as part of their infrastructure and what their contingency plan is. It turns out so far, there isn't one. Other than contacting the CEO of Cloudflare rather than switching on a temporary mitigation measure to ensure minimal downtime. Therefore, many engineers at affected companies would ha…

Sometimes it's not worth it. Your plan is just to accept you'll be off for a day or two, while you switch to a competitor.

If there's a fitting competitor worth switching to.

Plus most people don't get blamed when AWS (or to a lesser extent Cloudflare) goes down, since everyone knows more than half the world is down, so there's not an urgent motivation to develop multi-vendor capability.

Re: Cloudflare was down

#516

Post by tialaramex »

ethbr1 wrote: post #512
tialaramex wrote: post #488

Earlier quoted context omitted.

However, one of the things you see (if you read enough of them) in accident investigation reports for regulated industries is a recurring pattern 1. Accident happens 2. Investigators conclude Accident would not happen if people did X. Recommend regulator requires that people do X, citing previous such recommendations each iteration 3. Regulator declined this recommendation, arguing it's too expensive to do X, or peop…

As bad as (3) sounds, I'll strongman the argument: it's important to keep the economic cost of any regulation in mind.* On the one hand, you'd like to prevent the thing the regulation is seeking to prevent. On the other hand, you'd have costs for the regulation to be implemented (one-time and/or ongoing). "Is the good worth the costs?" is a question worth asking every time. (Not least because sometimes it lets you do…

I think conventionally the verb is "to steelman" with the intended contrast being to a strawman, an intentionally weak argument by analogy to how straw isn't strong but steel is. I understood what you meant by "strongman" but I think that "steelman" is better here.

There is indeed a good reason regulators aren't just obliged to institute all recommendations - that would be a lot of new rules. The only accident report I remember reading with zero recommendations was a MAIB (Maritime accidents) report here which concluded that a crew member of a fishing boat has died at sea after their vessel capsized because they both they and the skipper (who survived) were on heroin, the rationale for not recommending anything was that heroin is already illegal, operating a fishing boat while on heroin is already illegal, and it's also obviously a bad idea, so, there's nothing to recommend. "Don't do that".

Cost is rarely very persuasive to me, because it's very difficult to correctly estimate what it will actually cost to change something once you decided it's required - based on current reality where it is not. Mass production and clever cost reductions resulting from the normal commercial pressures tend to drive down costs when we require something but not before (and often not after we cease to require it either)

It's also difficult to anticipate all benefits from a good change without trying it. Lobbyists against a regulation will often try hard not to imagine benefits after all they're fighting not to be regulated. But once it's in action, it may be obvious to everyone that this was just a better idea and absurd it wasn't always the case.

Remember when you were allowed to smoke cigarettes on aeroplanes? That seems crazy, but at the time it was normal and I'm sure carriers insisted that not being allowed to do this would cost them money - and perhaps for a short while it did.

Re: Cloudflare was down

#517

Post by cryptonym »

tempaccount420 wrote: post #509
cryptonym wrote: post #501

Earlier quoted context omitted.

True, thanks for sharing. Worth mentioning that's on the "full-stack" part of the framework. It doesn't impact most React website while it impacts most next.js websites.

It was React. Code in React's repository had to be patched to fix this. Next.JS just happens to be the biggest user of this part of React, but blaming Next.JS is weird...

Thanks, that's what I acknowledged in the message you just replied to.

I'm not blaming anyone. Mostly outlining who was impacted as it's not really related to the front-end parts of the framework that the initial comment was referring to.

Re: Cloudflare was down

#518

Post by The_President »

yoctosec wrote: post #115

I use Cloudflare because of their Tunnel to protect my Raspberry Pi, but I think I will just use it without the Tunnel now. My main concern is privacy, but I'm not ready to accept so frequent downtime and dependence on them. The whole reason to host self-host was to be independent anyway. Does anyone have a recommendation for that (that is free)? Should I worry about privacy? My name and my city are on the website an…

You can basically do this yourself without punching holes out to the public. Create VPN for Pi and clients and access from the same private network.

Re: Cloudflare was down

#519

Post by o_m »

luastoned wrote: post #439

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com…

I wonder if this is the new normal? Weekly Cloudflare outages that breaks huge parts of the internet.

Re: Cloudflare was down

#520

Post by cryptonym »

rvz wrote: post #481
cryptonym wrote: post #306

Earlier quoted context omitted.

Sometimes it's not worth it. Your plan is just to accept you'll be off for a day or two, while you switch to a competitor.

Can't say that when it is a time critical service such as hospitals, banks, financial institutions or air-traffic control services.

Only a fool would build an architecture for critical air-traffic with Cloudflare as a SPoF.
Post reply on HN