Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud region currently down due to water intrusion

status.cloud.google.com

Post reply on HN
Print view View original

71–80 of 187 posts

Re: Google Cloud region currently down due to water intrusion

#71

Post by lamontcg »

dekhn wrote: post #50
t0mas88 wrote: post #46

I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.

zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.

I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.

Re: Google Cloud region currently down due to water intrusion

#72

Post by ironmagma »

DebtDeflation wrote: post #49
jacquesm wrote: post #7

This seems to significantly under-report what's going on, see: https://www.theregister.com/2023/04/26/google_cloud_outage/ There is mention of a fire as well.

Plot twist: the server racks were made out of sodium.

NaCl, the revolutionary Sodium Cloud technology.

Re: Google Cloud region currently down due to water intrusion

#77

Post by benatkin »

worldsavior wrote: post #70
benatkin wrote: post #63

Earlier quoted context omitted.

1) Pay for stuff 2) Not be able to use it 3) Company continues to pretend this doesn't happen on the regular

> Company continues to pretend this doesn't happen on the regular What do you want them to say? "Hey we have X breakdowns but please, pay!"

Admit that it's not really serverless or cloud computing but someone else's computers.

Or just cut down on nonsense in some way.

If they can figure out how to make nonsense, they can figure out how to dial it back.

Re: Google Cloud region currently down due to water intrusion

#78

Post by sbierwagen »

radicaldreamer wrote: post #56

Not sure what kind of fire there was there, but once those automatic sprinkler systems get going, they are very difficult to stop. Someone in my freshman college dorm decided to use one as a clothes hanger hook and broke the thermometer in there. The sprinkler damaged the entire floor with water and the floor below had spotty rain as well. The fire department came and was mainly concerned about evacuating everyone ra…

Always worth tracking down the sprinkler shut off valve in your residence/place of work. If you’re in a high rise it‘ll be the big red wheel on the sprinkler main in the fire stairs. If it’s a spurious activation you can just shut it off yourself, you don’t need to ask anybody’s permission.

The fire department is always going to prioritize safety of life, and after all it’s not their stuff getting soaked.

Re: Google Cloud region currently down due to water intrusion

#80

Post by londons_explore »

lamontcg wrote: post #71
dekhn wrote: post #50

Earlier quoted context omitted.

zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.

I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.

I believe the Google design is one big pool of machines, perhaps spread across a few buildings, but they hope that any failure only affects a few racks.

They will arrange/move workloads such that any one customer will only see an outage in one 'zone'.

Clearly that didn't work here,

Post reply on HN