Google Cloud Europe service disruption • Hacker News Board

Re: Google Cloud Europe service disruption

#111

Post by CydeWeys » Wed, Apr 26, 2023, 3:38 PM UTC

oblio wrote: post #72

hf_twink wrote: post #42

Earlier quoted context omitted.

that’s how GCP does zones, firewalled off with separate networks/power in the same physical location.

That's just ridiculous. AWS, for comparison: > AZs make partitioning applications for high availability easy. If an application is partitioned across AZs, companies are better isolated and protected from issues such as power outages, lightning strikes, tornadoes, earthquakes, and more. AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles)…

AWS has similarly suffered outages from an entire datacenter being taken out like this. No one is immune. If you want true fault-tolerance you need to be multi-regional (everyone says as much), ideally, multi-continental.

europe-west9 is the only large Google datacenter in France afaik. Building more would cost lots more money, and it seems like the market isn't there for it. Workloads that require data locality in France are presumably suffering the most. And there are knock-on effects on other datacenters from losing an entire huge chunk of capacity like this.

Re: Google Cloud Europe service disruption

#112

Post by stall84 » Wed, Apr 26, 2023, 3:41 PM UTC

rippercushions wrote: post #12

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

This isn't one of the under-ocean data-centers I've seen that (at least) Microsoft had been building in the Atlantic right? (They help with cooling, obviously if under ocean)

Re: Google Cloud Europe service disruption

#113

Post by electroly » Wed, Apr 26, 2023, 3:42 PM UTC

derefr wrote: post #75

londons_explore wrote: post #41

Earlier quoted context omitted.

> There is a separate issue stopping changes to HTTP load balancers across most of GCP Is it me, or has Google had issues with pushing changes to load balancers pretty much every few months for the past decade? Even before GCP launched, people here on HN sometimes said an outage was extended because load balancer configs couldn't be changed. Have they not considered just redesigning their config push mechanism...

My impression, from reading the docs around Google's "premium-tier network routing" — and just from the "feeling" of deploying GCLB updates — is that when you're configuring "a" Google Cloud Load Balancer, you're actually configuring "the" Google Cloud Load Balancer. I.e., your per-tenant virtual LB config resources, get baked down along with every other tenants' virtual LB config resources, to form a single real con…

AWS separates the anycast LB functionality into a separate service called AWS Global Accelerator. You do get individual anycast IP addresses with that service.

Re: Google Cloud Europe service disruption

#114

Post by bushbaba » Wed, Apr 26, 2023, 3:52 PM UTC

aeyes wrote: post #32

xrayarx wrote: post #3

Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region https://twitter.com/GCP_Incidents

But europe-west9-a is only one zone, why does the whole region fall over as a consequence?

GCP has multiple zones in the same physical building. Not all cloud providers have distinct physical buildings for each Availability Zone.

Re: Google Cloud Europe service disruption

#115

Post by derefr » Wed, Apr 26, 2023, 3:52 PM UTC

electroly wrote: post #113

derefr wrote: post #75

Earlier quoted context omitted.

My impression, from reading the docs around Google's "premium-tier network routing" — and just from the "feeling" of deploying GCLB updates — is that when you're configuring "a" Google Cloud Load Balancer, you're actually configuring "the" Google Cloud Load Balancer. I.e., your per-tenant virtual LB config resources, get baked down along with every other tenants' virtual LB config resources, to form a single real con…

AWS separates the anycast LB functionality into a separate service called AWS Global Accelerator. You do get individual anycast IP addresses with that service.

Ah, interesting; it's been a while since I played with AWS, and that service wasn't there back then. I'm guessing that allocating a new AWS Global Accelerator address takes a while?

Re: Google Cloud Europe service disruption

#116

Post by bushbaba » Wed, Apr 26, 2023, 3:53 PM UTC

CydeWeys wrote: post #111

oblio wrote: post #72

Earlier quoted context omitted.

That's just ridiculous. AWS, for comparison: > AZs make partitioning applications for high availability easy. If an application is partitioned across AZs, companies are better isolated and protected from issues such as power outages, lightning strikes, tornadoes, earthquakes, and more. AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles)…

AWS has similarly suffered outages from an entire datacenter being taken out like this. No one is immune. If you want true fault-tolerance you need to be multi-regional (everyone says as much), ideally, multi-continental. europe-west9 is the only large Google datacenter in France afaik. Building more would cost lots more money, and it seems like the market isn't there for it. Workloads that require data locality in F…

Eh source for that. AWS has had issues where a single Zone caused such a lack of capacity in the region that some multi-zone services degraded to the point of a domino fail-over. However I've not heard of any AWS event where a fire/flood in AZ A also caused a fire/flood in AZ B.

Re: Google Cloud Europe service disruption

#117

Post by miyuru » Wed, Apr 26, 2023, 3:57 PM UTC

derefr wrote: post #75

londons_explore wrote: post #41

Earlier quoted context omitted.

> There is a separate issue stopping changes to HTTP load balancers across most of GCP Is it me, or has Google had issues with pushing changes to load balancers pretty much every few months for the past decade? Even before GCP launched, people here on HN sometimes said an outage was extended because load balancer configs couldn't be changed. Have they not considered just redesigning their config push mechanism...

My impression, from reading the docs around Google's "premium-tier network routing" — and just from the "feeling" of deploying GCLB updates — is that when you're configuring "a" Google Cloud Load Balancer, you're actually configuring "the" Google Cloud Load Balancer. I.e., your per-tenant virtual LB config resources, get baked down along with every other tenants' virtual LB config resources, to form a single real con…

> CloudFront is anycast-routed

This is false, cloudfront uses DNS (geo & latency) based load balancing.

Re: Google Cloud Europe service disruption

#118

Post by electroly » Wed, Apr 26, 2023, 4:00 PM UTC

derefr wrote: post #115

electroly wrote: post #113

Earlier quoted context omitted.

AWS separates the anycast LB functionality into a separate service called AWS Global Accelerator. You do get individual anycast IP addresses with that service.

Ah, interesting; it's been a while since I played with AWS, and that service wasn't there back then. I'm guessing that allocating a new AWS Global Accelerator address takes a while?

I've only done it once (the way they have it architected, it's a "set and forget" sort of thing, your LB changes don't touch the Global Accelerator) but I do seem to recall that it took awhile to create the resource. Maybe 5-10 minutes?

Re: Google Cloud Europe service disruption

#119

Post by milesward » Wed, Apr 26, 2023, 4:07 PM UTC

tinco wrote: post #56

rippercushions wrote: post #12

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

So this is probably too soon, thoughts and prayers for the datacenter operators and staff out there, but are they going to auction off the flooded hardware? Trying to restore a flooded Google rack sounds like a super fun project. Anyone experience with losing an entire DC to flooding? edit: I just Googled it (lol) and this DC has to be brand spanking new ( https://cloud.google.com/blog/products/infrastructure/google.…

The machines are not industry standard stuff, and they don't auction, they destroy for customer security. See here: https://www.datacenterknowledge.com/google-alphabet/robots-n...

Re: Google Cloud Europe service disruption

#120

Post by milesward » Wed, Apr 26, 2023, 4:09 PM UTC

jsonne wrote: post #96

neeleshs wrote: post #79

Earlier quoted context omitted.

GCP doesn't operate the same way as Google consumer products. We are a paid customer for over 5 years and I also have only good things to say about GCP and their support

Really because I'm a GCP customer also and earlier this week they arbitrarily shut off Looker on us with no explanation leading to tons of pissed off customers. Our account executive responded with no help and a link to file a ticket. I expect a lot more from a service we're paying $10k+ a month for and my experience with Google has been so bad we're considering migrating everything to Microsoft.

Really? I'm happy to help: miles@sada.com