Live data from Hacker News

Google Cloud Europe service disruption

status.cloud.google.com

31–40 of 153 posts

Re: Google Cloud Europe service disruption

#31
post #16
post #7

Earlier quoted context omitted.

How large is the flood? How far away are these zones?

It happened at GlobalSwitch Clichy, near Paris. From what I gathered from a french forum[1], it started with a flood and then a fire. No rooms have been affected, apparently. [1]: https://lafibre.info/datacenter/incendie-maitrise-globalswit...

If it's the one in Clichy I'm thinking of it's dug into the embankment that lines a railway basin, so... yeah, floods suck.

Re: Google Cloud Europe service disruption

#32
post #3

Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region https://twitter.com/GCP_Incidents

But europe-west9-a is only one zone, why does the whole region fall over as a consequence?

Re: Google Cloud Europe service disruption

#33
post #32
post #3

Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region https://twitter.com/GCP_Incidents

But europe-west9-a is only one zone, why does the whole region fall over as a consequence?

Probably some dependencies they did not plan for

Re: Google Cloud Europe service disruption

#35
> Water intrusion in europe-west9-a

> We expect general unavailability of the europe-west9 region.

Why would emergency shutdown of a single AZ lead to general unavailability of a region? Isn't that the point of multiple AZs?

> There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage

yikes

Re: Google Cloud Europe service disruption

#36
post #32

Earlier quoted context omitted.

But europe-west9-a is only one zone, why does the whole region fall over as a consequence?

Probably some dependencies they did not plan for

Switching off all of 1 zone and checking the others aren't impacted is literally step 1 of checking your organisation is truly zonally redundant...

Someone as big as Google ought to have been practicing this automatically every week in a staging environment, and probably at least annually in production.

Re: Google Cloud Europe service disruption

#38

Earlier quoted context omitted.

At least Paris is red, what's the last time you saw more than a green dot with a little exclamation mark on the AWS status page?

I was referring to how things being “back to normal except for Paris” is a common occurrence here in Europe

They should just stick a big "on strike" banner across the status page...

Re: Google Cloud Europe service disruption

#39

> Water intrusion in europe-west9-a > We expect general unavailability of the europe-west9 region. Why would emergency shutdown of a single AZ lead to general unavailability of a region? Isn't that the point of multiple AZs? > There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage yikes

From other comments here, it sounds like multiple zones in that region are located in the same datacenter?

If so, that's ... not good.

Post reply on HN