Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Europe service disruption

status.cloud.google.com

Post reply on HN
Print view View original

101–110 of 153 posts

Re: Google Cloud Europe service disruption

#101

Post by Pr0Ger »

tinco wrote: post #56
rippercushions wrote: post #12

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

So this is probably too soon, thoughts and prayers for the datacenter operators and staff out there, but are they going to auction off the flooded hardware? Trying to restore a flooded Google rack sounds like a super fun project. Anyone experience with losing an entire DC to flooding? edit: I just Googled it (lol) and this DC has to be brand spanking new ( https://cloud.google.com/blog/products/infrastructure/google.…

A long time ago, one server room (located in the basement of the university building) of SPB-IX was flooded. It was a fun day for engineers whom unplugged survived equipment standing knee-deep in water

It was before dam (1) was built and floods were a huge problem in SPB

[1]: https://en.wikipedia.org/wiki/Saint_Petersburg_Dam

Re: Google Cloud Europe service disruption

#102

Post by numbsafari »

MisterTea wrote: post #86
numbsafari wrote: post #71

Earlier quoted context omitted.

I once was a customer of a DC who's roof drainage was clogged, turning it into a lake after a couple of rain storms. It then proceeded to rain inside the DC as the roof started to leak from all the pressure. "Servers are down, I'll head over to the DC" turned into "Um... it's raining _in the DC_. Get me some tarps and get us cut over to the backup in the office". Ah, the glory days of running out of a single co-lo ac…

As someone who has owned two commercial flat roof buildings I cant stress enough that you MUST do inspections of your roof at least twice a year. Especially if you live in a big city. I've had backups caused by kids roofing balls and bottles, stolen purse, dead squirrel, dirty balled up diapers from the neighboring apartment building. City living for ya.

Yeah, I'm pretty sure in this case it was a combination of having a 4ft parapet around the entire roof, and having basically never done an inspection. Not enough drains and they were all full of leaf matter.

Re: Google Cloud Europe service disruption

#104

Post by wkat4242 »

re-thc wrote: post #77
derefr wrote: post #75

Earlier quoted context omitted.

My impression, from reading the docs around Google's "premium-tier network routing" — and just from the "feeling" of deploying GCLB updates — is that when you're configuring "a" Google Cloud Load Balancer, you're actually configuring "the" Google Cloud Load Balancer. I.e., your per-tenant virtual LB config resources, get baked down along with every other tenants' virtual LB config resources, to form a single real con…

> It's intriguing to me that AFAIK load-balancer changes in AWS don't suffer from this problem. (Though, on the other hand, CDN changes do.) The architecture is a lot different. Using google means working with the load balancer in some form. It's all interconnected. AWS is all separate parts that are stitched together thinly. E.g. you can have a single global load balancer in Google that handles your whole infrastruc…

> AWS is all separate parts that are stitched together thinly.

Yeah I always hate this when I have to work with AWS. All their services feel like they were designed by completely different companies. Every management interface looks and feels different, and there are tons of services that do almost the same thing so it's not clear which would be best to use. It's a maze to me.

Luckily I don't have to work with cloud a lot but I really prefer Azure where everything is in the same console and there isn't a lot of overlap. But cloud guys seem to hate it, not sure why.

Re: Google Cloud Europe service disruption

#105

Post by wkat4242 »

tinco wrote: post #56
rippercushions wrote: post #12

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

So this is probably too soon, thoughts and prayers for the datacenter operators and staff out there, but are they going to auction off the flooded hardware? Trying to restore a flooded Google rack sounds like a super fun project. Anyone experience with losing an entire DC to flooding? edit: I just Googled it (lol) and this DC has to be brand spanking new ( https://cloud.google.com/blog/products/infrastructure/google.…

Umm thoughts and prayers? It's not as if their house is being washed away :) They just have a busy day at work. Keeps things exciting :P

Re: Google Cloud Europe service disruption

#106

Post by dannyw »

Xylakant wrote: post #88
antonvs wrote: post #85

Earlier quoted context omitted.

> a surprisingly small amount of water ingress would trip a breaker while leaving the racks in good order. If that were the case they wouldn't be saying "There is no current ETA for recovery," and "it is expected to be an extended outage. Customers are advised to failover to other regions."

Starting up a cloud region after a total shutdown is likely an untested procedure with no well known timeframe, even if the hardware is ok.

If you're in the business of being a massive cloud provider, hopefully restarting a region is not an untested procedure for you.

You could always test this in a live environment before a region becomes open to customers.

Re: Google Cloud Europe service disruption

#108

Post by CydeWeys »

antonvs wrote: post #85
pjc50 wrote: post #76

Earlier quoted context omitted.

I'm not sure what the disk encryption story is in Google Cloud but I'd rather it didn't end up on Ebay. Mind you, "flooded" covers a wide range of possibilities and a surprisingly small amount of water ingress would trip a breaker while leaving the racks in good order.

> a surprisingly small amount of water ingress would trip a breaker while leaving the racks in good order. If that were the case they wouldn't be saying "There is no current ETA for recovery," and "it is expected to be an extended outage. Customers are advised to failover to other regions."

There's a lot more to a datacenter building than just the servers sitting on racks. In particular here there was a fire in the power-serving infrastructure (caused by the flood presumably). So nearly all of those servers could be totally fine, just off, but if the power distribution network in the building is literally fried, that's gonna take a long time to fix.

Re: Google Cloud Europe service disruption

#109

Post by bushbaba »

tinco wrote: post #56
rippercushions wrote: post #12

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

So this is probably too soon, thoughts and prayers for the datacenter operators and staff out there, but are they going to auction off the flooded hardware? Trying to restore a flooded Google rack sounds like a super fun project. Anyone experience with losing an entire DC to flooding? edit: I just Googled it (lol) and this DC has to be brand spanking new ( https://cloud.google.com/blog/products/infrastructure/google.…

Likely not. It’s also not Google’s first dc flood/water intrusion causing a GCP incident.
Post reply on HN