Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud region currently down due to water intrusion

status.cloud.google.com

Post reply on HN
Print view View original

151–160 of 187 posts

Re: Google Cloud region currently down due to water intrusion

#151

Post by palcu »

Waterluvian wrote: post #142
palcu wrote: post #138

[disclaimer: SRE @ Google, I was involved with the incident, obvious conflicts of interest] Hey Dang, thanks for cleaning up the thread. One thing to note is that the title is not correct. The entire region is not currently down, as the regional impact was mitigated as of 06:39 PDT, per the support dashboard (though I think it was earlier). The impact is currently zonal (europe-west9-a), so having zone in the title a…

Would you be able to comment a bit on the emotional (perhaps there’s a better word) aspect of the response? Was there a lot of anxiety? Panic? Or was it just a “woof that sucks. Time to follow a checklist and then do a bunch of paper work” ? What I’m curious about is what it feels like on a team at a company like Google when there is a major system failure.

There's not much emotion as the core team working on the huge outages is more like an "SRE for SRE". They are all people who've been with the company for a long time and they've been in the secondary seat for at least one previous big rodeo. Not to mention that we're all running a checklist that has been exercised multiple times and there's always somebody on the call who could help if a step fails.

Personally, I wasn't part this time for the actual mitigation of the overall Paris DC recovery, as I was busy with an unfortunate[0] side effect of the outage. These generate more anxiety, as being woken up at 6am and being told that nobody understands exactly why the system is acting this way is not great. But then again, we're trained for this situation and there are always at least several ways of fixing the issue.

Finally, it's worth repeating that incident management is just a part of the SRE job and after several years I've understood that it is not the most important one. The best SREs I know are not great when it comes to a huge incident. But, they're work has avoided the other 99 outages that could have appeared on the front page of Hacker News.

[0]: https://news.ycombinator.com/item?id=35734224

Re: Google Cloud region currently down due to water intrusion

#152

Post by kyrra »

stingraycharles wrote: post #129
kyrra wrote: post #92

Earlier quoted context omitted.

Googler, opinions are my own. I think you misunderstand Google's infrastructure. I'm guessing that each GCP zone is actually a Borg Cell (see: https://storage.googleapis.com/pub-tools-public-publication-... ). Borg cells tend to be isolated from eachother in many ways in the physical layer (networking and management being a big one, not sure about power). So networking or machine management for an entire zone could g…

I don’t know what you’re trying to say with Borg cells, the point of discussion is not that the network etc are separated, but that they’re physically separated in such a way that these kind of flooding wouldn’t affect different AZs, and that GCP is cutting corners here. Obviously every cloud vendor recommends replicating data between multiple regions, but fact of the matter is that a lot of cloud services work much…

I guess it's different types of concerns. My feeling is that Google tries to optimize the resources of a datacenter, and the larger it is, the better things can scale. GCP Zones provide logical separation of machines for management (and network). There may be physical separation, but within a given region, GCP does not advertise this.

I think Google designs their datacenters for their own needs and expect you (a product running in their DCs) to distribute by region. Almost products at Google will be operating in multiple regions given the reach of most of our services, so DC design followed that need.

Based on GCP's docs, they still think region separate is better. Not sure why you wouldn't just do that?

If there is a catastrophic event (a large tornado hit AWS us-east-2), those buildings are pretty close to one another and both likely would be taken out, right? So you could lose multiple AZs since they are physically located so close to one another?

Re: Google Cloud region currently down due to water intrusion

#153

Post by deagle50 »

radicaldreamer wrote: post #56

Not sure what kind of fire there was there, but once those automatic sprinkler systems get going, they are very difficult to stop. Someone in my freshman college dorm decided to use one as a clothes hanger hook and broke the thermometer in there. The sprinkler damaged the entire floor with water and the floor below had spotty rain as well. The fire department came and was mainly concerned about evacuating everyone ra…

This happened in my freshman dorm as well. The broken sprinkler was on the 3rd or 4th floor and my room which was on the 1st got at least an inch of water.

Re: Google Cloud region currently down due to water intrusion

#154

Post by jamesfinlayson »

mvanbaak wrote: post #117
radicaldreamer wrote: post #56

Not sure what kind of fire there was there, but once those automatic sprinkler systems get going, they are very difficult to stop. Someone in my freshman college dorm decided to use one as a clothes hanger hook and broke the thermometer in there. The sprinkler damaged the entire floor with water and the floor below had spotty rain as well. The fire department came and was mainly concerned about evacuating everyone ra…

datacenters dont use sprinkler systems (or at least they should not).

I have no idea what's used these days but I remember someone telling me the dry powder that is sometimes used is very bad for computer hardware - not immediately, but within a couple of years the metal will show obvious signs of reaction with whatever is in it.

Re: Google Cloud region currently down due to water intrusion

#155

Post by jamesfinlayson »

kevan wrote: post #25
mcast wrote: post #12

Earlier quoted context omitted.

AWS treats its availability zones very seriously, each zone has its own independent power substation, air conditioning, and fiber lines. It's incredibly rare for multiple AZs to go down at once, especially since they are more than a few miles apart from each other.

Funnily enough floods (GCP) and fires (OVH) are two of the 3 things AWS explicitly mentions in the Well Architected docs. For a lot of companies an AZ going down is an annoyance or bad day but a whole region going down could be a real continuity risk. > Each Availability Zone is separated by a meaningful physical distance from other zones to avoid correlated failure scenarios due to environmental hazards like fires,…

> but a whole region going down could be a real continuity risk

Very much so - Australia only got a second region this year, so if your work required data to remain in Australia, you just had to hope that ap-southeast-2 didn't have a major issues. I'm sure there are plenty of other countries with only a single region.

Re: Google Cloud region currently down due to water intrusion

#156

Post by jamesfinlayson »

asymptotic wrote: post #128

When I worked at AWS there was a similar scenario in eu-west-2. There was a fire in one of the availability zones (AZs). The fire suppression system kicked in and flooded the data center up to ankle or knee height. All the racks were powered off and the building was evacuated for hours (I don't remember the duration of the evacuation) until the water was pumped out. But for the service team I worked for, our AZ-evacu…

> urprisingly many service teams at AWS still operate services that don't handle AZ failure that well (at the time)

Ouch, hopefully none of the major services? I recently had to look into this for work (for disaster recovery preparation) and it seemed like ECS, Lambda, S3, DynamoDB and Aurora Serverless (and probably CloudWatch and IAM) all said they handled availability zone failures transparently enough.

Re: Google Cloud region currently down due to water intrusion

#157

Post by Aeolun »

local_crmdgeon wrote: post #118
influx wrote: post #91

Earlier quoted context omitted.

AWS has had datacenters that were flooded, but they failed over and physically moved racks via trucks. Customers never knew.

This has happened much more than people know.

If it happens without me knowing then they’re doing their job very well.

Re: Google Cloud region currently down due to water intrusion

#158

Post by Aeolun »

packetslave wrote: post #141
mvanbaak wrote: post #117

Earlier quoted context omitted.

datacenters dont use sprinkler systems (or at least they should not).

A non-water fire suppression system for a 300,000+ square feet warehouse-scale datacenter would be incredibly expensive.

Would it be as expensive as replacing 300,000+ square feet of servers?

Re: Google Cloud region currently down due to water intrusion

#159

Post by Waterluvian »

palcu wrote: post #151
Waterluvian wrote: post #142

Earlier quoted context omitted.

Would you be able to comment a bit on the emotional (perhaps there’s a better word) aspect of the response? Was there a lot of anxiety? Panic? Or was it just a “woof that sucks. Time to follow a checklist and then do a bunch of paper work” ? What I’m curious about is what it feels like on a team at a company like Google when there is a major system failure.

There's not much emotion as the core team working on the huge outages is more like an "SRE for SRE". They are all people who've been with the company for a long time and they've been in the secondary seat for at least one previous big rodeo. Not to mention that we're all running a checklist that has been exercised multiple times and there's always somebody on the call who could help if a step fails. Personally, I was…

I appreciate your insight into this. Thanks!

Re: Google Cloud region currently down due to water intrusion

#160

Post by BlackjackCF »

lamontcg wrote: post #73

Nobody here with any thoughts for the operations/datacenter engineers trying to deal with stopping and cleaning up the disaster, just customers complaining...

I think people who are complaining are stressed out about their own services being down.

If you’ve only ever used the cloud, you’re not necessarily aware of everything that’s involved at data centers. If you’re not familiar with them, I don’t think you’d know how many things can (literally) blow up in your face. If someone sees flooding, they generally aren’t thinking that it’ll lead to fires.

Anyway, just want to think that everyone generally has good intentions and just don’t know what’s ACTUALLY happening in the DC, or how much work it will be for the folks working in the DC to restore services.

Hopefully all the failsafes kicked in and worked and nobody was injured.

Post reply on HN