Live data from Hacker News

Google Cloud Europe service disruption

status.cloud.google.com

51–60 of 153 posts

Re: Google Cloud Europe service disruption

#51

Earlier quoted context omitted.

From other comments here, it sounds like multiple zones in that region are located in the same datacenter? If so, that's ... not good.

that’s how GCP does zones, firewalled off with separate networks/power in the same physical location.

Are you joking? Please tell me that’s a joke, because there’s no way a cloud provider that big could be that daft.

If that’s true, what’s the fucking point of separating them at all?

Re: Google Cloud Europe service disruption

#54
post #51

Earlier quoted context omitted.

that’s how GCP does zones, firewalled off with separate networks/power in the same physical location.

Are you joking? Please tell me that’s a joke, because there’s no way a cloud provider that big could be that daft. If that’s true, what’s the fucking point of separating them at all?

Up-sell.

Re: Google Cloud Europe service disruption

#55

Earlier quoted context omitted.

Ouch. Isn't part of separate zones being protected against something, say, like a terrorist attack or a natural disaster that can take down a whole datacenter?

AFAIK if you dig into the details, the different cloud providers have very different concepts of what constitutes an AZ with respect to the types of faults that are isolated.

I always felt a bit scammed with AWS Multi-AZ on RDS that basically doubles your cost. If their set up is anything like this, I now feel vindicated in turning it off....

Re: Google Cloud Europe service disruption

#56

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

So this is probably too soon, thoughts and prayers for the datacenter operators and staff out there, but are they going to auction off the flooded hardware? Trying to restore a flooded Google rack sounds like a super fun project.

Anyone experience with losing an entire DC to flooding?

edit: I just Googled it (lol) and this DC has to be brand spanking new (https://cloud.google.com/blog/products/infrastructure/google...), apparently they just opened it last June. Google must be livid with the contractors who built the place for it to get flooded so soon.

Re: Google Cloud Europe service disruption

#57

Earlier quoted context omitted.

AFAIK if you dig into the details, the different cloud providers have very different concepts of what constitutes an AZ with respect to the types of faults that are isolated.

I always felt a bit scammed with AWS Multi-AZ on RDS that basically doubles your cost. If their set up is anything like this, I now feel vindicated in turning it off....

It isn’t like this. AWS Availability Zones are in separate physical facilities by design, regardless of region.

https://aws.amazon.com/about-aws/global-infrastructure/regio...

Re: Google Cloud Europe service disruption

#58
Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

Re: Google Cloud Europe service disruption

#59
post #43
post #16

Earlier quoted context omitted.

It happened at GlobalSwitch Clichy, near Paris. From what I gathered from a french forum[1], it started with a flood and then a fire. No rooms have been affected, apparently. [1]: https://lafibre.info/datacenter/incendie-maitrise-globalswit...

I'm getting horrible flashbacks of OVH DC's those many years ago.

We were impacted at a previous company, luckily we had solid backup, so everything was back online a few hours after.

Still, it was kinda fun to go to work and learn that the corporate website literally went up in flames.

Re: Google Cloud Europe service disruption

#60

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

For some reasons that might be related to the 2nd issue, even though it says resolved, I am still seeing network errors in GKE nodes, located in Singapore (asia-southeast1)

  Warning  FailedToCreateRoute      4m59s                  route_controller  Could not create route fc61a148-b428-43fa-xxxx-xxxx 10.28.167.0/24 for node gke-xxx-xxx after 16.320065487s: googleapi: Error 503: INTERNAL_ERROR - Internal error. Please try again or contact Google Support.
Any facing something similar?
Post reply on HN