Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud region currently down due to water intrusion

status.cloud.google.com

Post reply on HN
Print view View original

171–180 of 187 posts

Re: Google Cloud region currently down due to water intrusion

#171

Post by jacquesm »

madaxe_again wrote: post #168
jacquesm wrote: post #137

Earlier quoted context omitted.

There's that beautiful German word again... schadenfreude. I have had similar discussions multiple times in the last year and the magic thinking around the cloud is so strong that it is sometimes impossible to get through. The fact that cloud stuff can go down and that in the end it is your data and no amount of cloud credits are going to help you if your data is lost seems to be utterly beyond some people's comprehe…

In this case it wasn’t so much “the cloud is invincible!” as “this appears to be a cost centre” - their whole shtick is to boost profitability in the short term by gutting businesses, and then selling them onwards to some other finance sucker. I am sure that my name is currently being damned in a boardroom in Chicago, as as the person who warned of this, I am likely seen as responsible.

> I am sure that my name is currently being damned in a boardroom in Chicago, as as the person who warned of this, I am likely seen as responsible.

In that sense nothing ever changes. Anyway, kudos to you for staying the course on topics like these.

Re: Google Cloud region currently down due to water intrusion

#172

Post by CyberDildonics »

samstave wrote: post #87
lukax wrote: post #13

Google now has a "data lake" in Paris.

Funny enough, Lucas Film suffered a same outage because their data center was backed up to the lake, and was basically under water, and then the wall started to leak.....

Do you have a source for that?

Re: Google Cloud region currently down due to water intrusion

#173

Post by throwbigdata »

palcu wrote: post #151
Waterluvian wrote: post #142

Earlier quoted context omitted.

Would you be able to comment a bit on the emotional (perhaps there’s a better word) aspect of the response? Was there a lot of anxiety? Panic? Or was it just a “woof that sucks. Time to follow a checklist and then do a bunch of paper work” ? What I’m curious about is what it feels like on a team at a company like Google when there is a major system failure.

There's not much emotion as the core team working on the huge outages is more like an "SRE for SRE". They are all people who've been with the company for a long time and they've been in the secondary seat for at least one previous big rodeo. Not to mention that we're all running a checklist that has been exercised multiple times and there's always somebody on the call who could help if a step fails. Personally, I was…

Who trains the trainers?

Re: Google Cloud region currently down due to water intrusion

#174

Post by hoofhearted »

hoofhearted wrote: post #147
H8crilA wrote: post #93

Earlier quoted context omitted.

You're not far off: the batteries are (probably) made of lithium. Also, why batteries in a datacenter? When you implement a flush() command at the lowest level you're faced with two choices: 1) actually write to disk, then return from the call, 2) write to some cache/RAM and have just enough battery locally to ensure that you can write it to disk even if all power goes out. Then there's the other problem of surviving…

The batteries in the datacenter are simply there to hold the power until the generators are all up and running, and the phases are in sync. They create 3 separate arrays of batteries in each back. Each array represents a power phase, A-B-C.. if I remember correctly, each array has a number of low voltage/2000 amp batteries connected in series to make up for a 2000amp 480 volt leg on the other end. In a tier 4 plus+1…

I wish I understood why my apple autocorrect stopped working lol

Re: Google Cloud region currently down due to water intrusion

#175

Post by sitkack »

deathanatos wrote: post #94
dekhn wrote: post #50

Earlier quoted context omitted.

zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.

> A cluster represents a distinct physical infrastructure that is housed in a data center. > Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages And they have stated that the flood "caused a multi-cluster failure". > Zones should be considered a single failure domain within a region. (—GCP's documentation.)

"should be considered" -> we would like you to think

Zones are an artificial construct.

Re: Google Cloud region currently down due to water intrusion

#177

Post by asymptotic »

jamesfinlayson wrote: post #156
asymptotic wrote: post #128

When I worked at AWS there was a similar scenario in eu-west-2. There was a fire in one of the availability zones (AZs). The fire suppression system kicked in and flooded the data center up to ankle or knee height. All the racks were powered off and the building was evacuated for hours (I don't remember the duration of the evacuation) until the water was pumped out. But for the service team I worked for, our AZ-evacu…

> urprisingly many service teams at AWS still operate services that don't handle AZ failure that well (at the time) Ouch, hopefully none of the major services? I recently had to look into this for work (for disaster recovery preparation) and it seemed like ECS, Lambda, S3, DynamoDB and Aurora Serverless (and probably CloudWatch and IAM) all said they handled availability zone failures transparently enough.

I’m familiar with Lambda and DynamoDB. When I left in 2022 they both had strong automated or semi-automated AZ evacuation stories.

I’m not that familiar with S3, but I never noticed any concerns with S3 during an AZ outage. I’m not at all familiar with Aurora Serverless or ECS.

For all AWS services you can always ask AWS Support pointed, specific questions about availability. They usually defer to the service team and they’ll give you their perspective.

Also keep in mind that AWS teams differentiate between the availability of their control and data planes. During an AZ outage you may struggle to create/delete resources until an AZ evacuation is completed internally, but already created resources should always meet the public SLA. That’s why especially for DR I recommend active-active or active-“pilot light”, have everything created in all AZs/regions and don’t need to create resources in your DR plan.

Re: Google Cloud region currently down due to water intrusion

#178

Post by joelrwilliams1 »

local_crmdgeon wrote: post #126
sokoloff wrote: post #19

Earlier quoted context omitted.

Unless they’re in us-east-1 and it’s an Amazon software/service fault.

This. Don't use us-east-1, it's by far the flakiest. PDX is also a bit rough, but Ohio is golden.

Ohio has tons of problems, no one should ever put their infra in us-east-2 (shhhhhh...don't let the secret out )

Re: Google Cloud region currently down due to water intrusion

#179

Post by palcu »

throwbigdata wrote: post #173
palcu wrote: post #151

Earlier quoted context omitted.

There's not much emotion as the core team working on the huge outages is more like an "SRE for SRE". They are all people who've been with the company for a long time and they've been in the secondary seat for at least one previous big rodeo. Not to mention that we're all running a checklist that has been exercised multiple times and there's always somebody on the call who could help if a step fails. Personally, I was…

Who trains the trainers?

Life and experience, if you're looking for a short answer. For example, last year we had an outage in London[0] and the folks who worked on it learnt a lot. Now, they applied the learnings in this incident.

[0]: https://news.ycombinator.com/item?id=32161755

Re: Google Cloud region currently down due to water intrusion

#180

Post by local_crmdgeon »

sbierwagen wrote: post #164
local_crmdgeon wrote: post #122

Earlier quoted context omitted.

Firefighters are personally incentivized to take photos like this: https://nypost.com/2022/01/10/fireman-in-post-photo-recalls-... . They are not incentivized to protect your stuff, that's what insurance does. Good. I want you to save me and my family from dying in a fucking fire. This thread is obsessed with saving Funko Pop collections for some reason??

Hello 16 day old account. Can't imagine why you've been banned before. https://news.ycombinator.com/item?id=35714352

I make many accounts because I don't like people snooping through my history. The Redditism of "I've looked back 10 years in your history, and will dismiss your comment because of something you said in 2015" is not a discussion I'm interested in.
Post reply on HN