Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud region currently down due to water intrusion

status.cloud.google.com

Post reply on HN
Print view View original

181–187 of 187 posts

Re: Google Cloud region currently down due to water intrusion

#181

Post by samstave »

CyberDildonics wrote: post #172
samstave wrote: post #87

Earlier quoted context omitted.

Funny enough, Lucas Film suffered a same outage because their data center was backed up to the lake, and was basically under water, and then the wall started to leak.....

Do you have a source for that?

Yeah, me. I was one of the lead designers on the Lucas Presidio campus, including the DC there...

When we were still designing the site/systems/infra we would go to BigRock Ranch a lot for meetings and vendor interviews.

Its the datacenter at bigrock which has one wall which backups to the lake on site, and the DC is in the sub-level parking garage, and its rear wall was leaking due to the pressure from the lake on the wall.

Re: Google Cloud region currently down due to water intrusion

#182

Post by CyberDildonics »

samstave wrote: post #181
CyberDildonics wrote: post #172

Earlier quoted context omitted.

Do you have a source for that?

Yeah, me. I was one of the lead designers on the Lucas Presidio campus, including the DC there... When we were still designing the site/systems/infra we would go to BigRock Ranch a lot for meetings and vendor interviews. Its the datacenter at bigrock which has one wall which backups to the lake on site, and the DC is in the sub-level parking garage, and its rear wall was leaking due to the pressure from the lake on t…

Was it really an 'outage' or did they just turn things off while they plugged the leak?

Re: Google Cloud region currently down due to water intrusion

#183

Post by samstave »

CyberDildonics wrote: post #182
samstave wrote: post #181

Earlier quoted context omitted.

Yeah, me. I was one of the lead designers on the Lucas Presidio campus, including the DC there... When we were still designing the site/systems/infra we would go to BigRock Ranch a lot for meetings and vendor interviews. Its the datacenter at bigrock which has one wall which backups to the lake on site, and the DC is in the sub-level parking garage, and its rear wall was leaking due to the pressure from the lake on t…

Was it really an 'outage' or did they just turn things off while they plugged the leak?

You know what would be an interesting article would be to have a ton of outage/post=mortems etc.

So some of mine woule be

Chinese air usb gaps and phishing

What stories can you expound upon?

Re: Google Cloud region currently down due to water intrusion

#184

Post by samstave »

CyberDildonics wrote: post #182
samstave wrote: post #181

Earlier quoted context omitted.

Yeah, me. I was one of the lead designers on the Lucas Presidio campus, including the DC there... When we were still designing the site/systems/infra we would go to BigRock Ranch a lot for meetings and vendor interviews. Its the datacenter at bigrock which has one wall which backups to the lake on site, and the DC is in the sub-level parking garage, and its rear wall was leaking due to the pressure from the lake on t…

Was it really an 'outage' or did they just turn things off while they plugged the leak?

Is your data center suffering an outage due to flooding from a faulty wall you built against the man-made lake you built next to your $ Billion data center??

Call California and Meyers to see if you qualify for PG&E benefits.

Re: Google Cloud region currently down due to water intrusion

#185

Post by illumin8 »

jacquesm wrote: post #137
madaxe_again wrote: post #135

Earlier quoted context omitted.

Oh, the irony. A few years ago I implemented a top to bottom ISO27k1 ISMS for a client handling extremely sensitive and mission-critical data for industry. One risk I recommended controls for was that of a fire and/or flood at their primary datacentre for their client-facing offerings - this datacentre. I’ve experienced the misery of a datacentre oops myself, firsthand, twice, and it’s a genuine risk that has to be m…

There's that beautiful German word again... schadenfreude. I have had similar discussions multiple times in the last year and the magic thinking around the cloud is so strong that it is sometimes impossible to get through. The fact that cloud stuff can go down and that in the end it is your data and no amount of cloud credits are going to help you if your data is lost seems to be utterly beyond some people's comprehe…

This really has nothing to do with cloud and is more of an "all eggs in one basket" problem. I wish people would stop painting cloud itself as less capable.

The fact is, most cloud providers offer multiple regions, which have the capability of giving you more geographic redundancy than most companies that operate in their own datacenters have.

Whether you choose to adopt a multi-region or multi-datacenter architecture is really orthogonal to whether you choose cloud or on-prem.

Re: Google Cloud region currently down due to water intrusion

#186

Post by illumin8 »

deathanatos wrote: post #94
dekhn wrote: post #50

Earlier quoted context omitted.

zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.

> A cluster represents a distinct physical infrastructure that is housed in a data center. > Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages And they have stated that the flood "caused a multi-cluster failure". > Zones should be considered a single failure domain within a region. (—GCP's documentation.)

I guess you just discovered the difference between a GCP Zone and an AWS Availability Zone.

By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains.

In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.

Re: Google Cloud region currently down due to water intrusion

#187

Post by jamesfinlayson »

asymptotic wrote: post #177
jamesfinlayson wrote: post #156

Earlier quoted context omitted.

> urprisingly many service teams at AWS still operate services that don't handle AZ failure that well (at the time) Ouch, hopefully none of the major services? I recently had to look into this for work (for disaster recovery preparation) and it seemed like ECS, Lambda, S3, DynamoDB and Aurora Serverless (and probably CloudWatch and IAM) all said they handled availability zone failures transparently enough.

I’m familiar with Lambda and DynamoDB. When I left in 2022 they both had strong automated or semi-automated AZ evacuation stories. I’m not that familiar with S3, but I never noticed any concerns with S3 during an AZ outage. I’m not at all familiar with Aurora Serverless or ECS. For all AWS services you can always ask AWS Support pointed, specific questions about availability. They usually defer to the service team an…

Okay good to know - Lambda seemed to suggest it could handle an availability zone going down without any trouble.

ECS Fargate's default is to distribute task instances across availability zones too but I'm assume if you use EC2 it might not be as straight-forward.

And that makes sense - I remember during the last outage that affected me it was a compute rather than data failure and the running stuff continued fine, just nothing new was getting created.

Post reply on HN