Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCP Outage

status.cloud.google.com

Post reply on HN
Print view View original

531–539 of 539 posts

Re: GCP Outage

#531

Post by craigseeman »

smoe wrote: post #377
whatevertrevor wrote: post #233

Earlier quoted context omitted.

Doesn't cloudflare have its own infrastructure, it's wild to me that both these things are down presumably together with this size of a blast radius.

Latest Cloudflare status update basically confirms that there is a dependency to GCP in their systems: "Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable"

They lightly mentioned it in this interview a few weeks ago as well - I was surprised! https://youtu.be/C5-741uQPVU?t=1726s

Re: GCP Outage

#532

Post by togume »

augbog wrote: post #382

Cloudflare Outage also just updated > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information

Is GCP the third party?

Re: GCP Outage

#533

Post by refulgentis »

9rx wrote: post #530
refulgentis wrote: post #525

Earlier quoted context omitted.

Because "It's good to lie because it makes people correct me" is a joke about IRC, not a viable stable game-theoretic optimal position.

Cunningham's Law emerged in the newsgroups era, well predating the existence of IRC. Of course, I recognize that you purposefully pulled the Cunningham's Law trigger so that you, too, would gain additional knowledge that nobody would have told you about otherwise, as one logically would. And that you played it off as some kind of derision towards doing that all while doing it yourself made it especially funny. Well d…

I have 0 idea what Cunningham's Law is, so we can both agree that "recognizing purpose" was "mind-reading", in this case. I didn't really bother reading the rest after the first sentence because I saw something about how I joking and congratulating me in my peripheral vision.

It is what it says on the tin: choosing to lie doesn't mean you want the truth communicated.

I apologize that it comes across as aggro, its just that I'm not quite as giggly about this as you are. I think I can safely assume you're old enough to recognize some deleterious effects of lying

Re: GCP Outage

#534

Post by 9rx »

refulgentis wrote: post #533
9rx wrote: post #530

Earlier quoted context omitted.

Cunningham's Law emerged in the newsgroups era, well predating the existence of IRC. Of course, I recognize that you purposefully pulled the Cunningham's Law trigger so that you, too, would gain additional knowledge that nobody would have told you about otherwise, as one logically would. And that you played it off as some kind of derision towards doing that all while doing it yourself made it especially funny. Well d…

I have 0 idea what Cunningham's Law is, so we can both agree that "recognizing purpose" was "mind-reading", in this case. I didn't really bother reading the rest after the first sentence because I saw something about how I joking and congratulating me in my peripheral vision. It is what it says on the tin: choosing to lie doesn't mean you want the truth communicated. I apologize that it comes across as aggro, its jus…

> I have 0 idea what Cunningham's Law is

You had no idea what it is. Now you know thanks to you the lie you told.

> choosing to lie doesn't mean you want the truth communicated.

But you're going to get it either way, so if you do lie, expect it. If you don't want it – don't lie, I guess. It is inconceivable that someone wouldn't want to learn about the truth, though. Sadly, despite your efforts in enacting Cunningham again, I don't have more information to give you here.

> I apologize that it comes across as aggro

It doesn't. Attaching human attributes to software would be plain weird.

> I think I can safely assume you're old enough to recognize some deleterious effects of lying

Time and place. While it can be disastrous in the right context, context is significant. It makes no difference in a place of entertainment, as is the case here. Entertainment has always been rooted in tales that aren't true. No matter how old you are, even young children understand that.

Re: GCP Outage

#535

Post by __turbobrew__ »

syllogism wrote: post #526
__turbobrew__ wrote: post #521

Earlier quoted context omitted.

The trick there is you take the relevant configs and serialize them to disk periodically, and then in a bootstrap scenario you use the configs on disk. Presumably for the infrequently read configs you could do this so the service with frequently read configs can bootstrap without the service for infrequently read configs.

Like a backup generator for inputs. Makes sense.

Yes, this is how I have set up systems to bootstrap.

For example a service discovery system periodically serializes peers to disk, and then if the whole thing falls down we have static IP addresses for a node and the service discovery system can use the last known IPs of peers to bring itself back up.

Re: GCP Outage

#536

Post by mnordhoff »

tfsh wrote: post #139
9rx wrote: post #43

Earlier quoted context omitted.

The program that updates the status page is hosted on Google Cloud.

It's not. You might be joking, but that comment still isn't helpful. My understanding is this is part of Google's internal PSD offering (Public Status Board) which uses SCS (Static Content Service) behind GFE (Google Frontend) which is hosted on Borg, and deploys other large scale apps such as Search, Drive, YouTube, etc.

Wellp. Incident report: "We posted our first incident report to Cloud Service Health about ~1h after the start of the crashes, due to the Cloud Service Health infrastructure being down due to this outage."

Re: GCP Outage

#537

Post by praveen4463 »

rvnx wrote: post #190
mrGomesDev wrote: post #178

Earlier quoted context omitted.

I use Expo intermediation for notifications, but with this Google context, I imagine that FCM is also suffering, is that possible?

Very likely. Firebase Auth is down for sure (though unreported yet), so most likely FCM too

whole of firebase was actuallky down

Re: GCP Outage

#538

Post by JohnMakin »

ManBeardPc wrote: post #517
asim wrote: post #439

Just our bi-yearly reminder of our over reliance on cloud providers for literally everything. Can't say there's an answer beyond trying to build more independent tech but we know how that goes.

Yet migration to the cloud continues, driven by people arguing that doing it yourself is too complicated and expensive. Let’s see how long until one outage takes down the global economy for multiple days or weeks.

"The cloud" isn't the problem nor is migration to it. The problem is single points of failure in a vast, super-connected, global network.

To demonstrate my point, say someone like cloudflare opted to get "off cloud" and run their own datacenters. Half the web would still go down if they had some issue in their datacenters. If anything, the economy of scale and resiliency of a huge cloud network is far beyond what any single operator can ensure for their own service. If this wasn't the case, cloud services couldn't be as profitable as they are.

It isn't the panacea people seem to think it is. One critical service going down, regardless of whether it's cloud hosted or not, has ripple effects in the broader network.

Re: GCP Outage

#539

Post by fruit_snack »

__turbobrew__ wrote: post #520
voytec wrote: post #388

Earlier quoted context omitted.

> outage of a 3rd party service that is a key dependency. Good to know that Cloudflare has services seemingly based on GCP with no redundancy.

After reading about cloudflare infra in post mortems it has always been surprising how immature their stack is. Like they used to run their entire global control plane in a single failure domain. Im not sure who is running the show there, but the whole thing seems kinda shoddy given cloudflares position as the backbone of a large portion of the internet. I personally work at a place with less market cap than cloudfla…

Nice job keeping your app up during the outage but I'm not sure you can say "the whole thing seems kinda shoddy" when they're handling the amount of traffic they are.
Post reply on HN