Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Europe service disruption

status.cloud.google.com

Post reply on HN
Print view View original

91–100 of 153 posts

Re: Google Cloud Europe service disruption

#91

Post by superzamp »

neeleshs wrote: post #79
tvanantwerp wrote: post #49

Earlier quoted context omitted.

This was my first thought too. Shows how Google has trained us to expect the worst from them...

GCP doesn't operate the same way as Google consumer products. We are a paid customer for over 5 years and I also have only good things to say about GCP and their support

I'm curious about how is the support these days. My last interaction with it was along the lines of "Uh, why are you contacting us for advice and not like, use the docs". This was for a new project and asking the same set of questions to AWS led them to send three guys to our office the next week for a round of demos and best-practices discussions.

Re: Google Cloud Europe service disruption

#92

Post by compumike »

rippercushions wrote: post #12

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

Wow, “physically flooded with water somehow” and “load balancers” config propagation issue are so drastically different!

Good reminder that downtime happens for many wild reasons, and you may want to take 30 seconds and set up a free website / API monitor with Heii On-Call [1] because we would have alerted you to either of these issues if they affected your app.

Really, a simple HTTP probe provides tremendous monitoring power. I already was telling people that it covered issues at the DNS, TCP, SSL certificate, load balancer, framework, and application layers. Now I will have to add “datacenter flood” as well :P

Best wishes to everyone working on europe-west-9.

[1] https://heiioncall.com/ (I recently helped build our HTTP probe background infrastructure in Crystal)

Re: Google Cloud Europe service disruption

#93

Post by briffle »

petedoyle wrote: post #74
mschuster91 wrote: post #46

Earlier quoted context omitted.

Ouch. Isn't part of separate zones being protected against something, say, like a terrorist attack or a natural disaster that can take down a whole datacenter?

I recently drove by both the GCP and AWS regions in Oregon. It was so interesting to see one giant facility for GCP, and like 40 separate datacenter buildings for AWS, typically separated by at least half a mile, sometimes tens of miles.

There are 2 buildings there for Google, serving 3 cloud zones. One of those buildings was google's first datacenter, so used some older ideas.

They are actually in the process of building 3 more buildings a ways down the road for more capacity.

Re: Google Cloud Europe service disruption

#94

Post by Kuinox »

neeleshs wrote: post #79
tvanantwerp wrote: post #49

Earlier quoted context omitted.

This was my first thought too. Shows how Google has trained us to expect the worst from them...

GCP doesn't operate the same way as Google consumer products. We are a paid customer for over 5 years and I also have only good things to say about GCP and their support

Meh, I seen multiples times peoples reporting here on hn, allegedly paying 5 figures in GCP and still having very bad customer support.

Re: Google Cloud Europe service disruption

#95

Post by AlfeG »

bsdz wrote: post #19
rippercushions wrote: post #12

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

I'm not sure if it's a separate issue but I've had trouble creating new VM instances in Google Cloud Console or listing GPU types using their CLI and I'm in europe-west-2. The ticket I was following originally got merged with the Paris flood ticket (by Google). It was working until midnight (London) last night but went down before 8am before recovering about 1h ago for me. Not sure why an outage at one regional data…

Every customer of affected region try to restore data/compute in other regions. It's quite known and expected issue in case of region loss.

Re: Google Cloud Europe service disruption

#96

Post by jsonne »

neeleshs wrote: post #79
tvanantwerp wrote: post #49

Earlier quoted context omitted.

This was my first thought too. Shows how Google has trained us to expect the worst from them...

GCP doesn't operate the same way as Google consumer products. We are a paid customer for over 5 years and I also have only good things to say about GCP and their support

Really because I'm a GCP customer also and earlier this week they arbitrarily shut off Looker on us with no explanation leading to tons of pissed off customers. Our account executive responded with no help and a link to file a ticket. I expect a lot more from a service we're paying $10k+ a month for and my experience with Google has been so bad we're considering migrating everything to Microsoft.

Re: Google Cloud Europe service disruption

#97

Post by ZeroCool2u »

neuronexmachina wrote: post #81
davidkuennen wrote: post #66

Earlier quoted context omitted.

TBH, GCP isn't operating like Google does. I'm a long time customer and have only good things to tell so far.

There's some exceptions, like Google Cloud Debugger which is getting shut down at the end of the month (although to be fair there was a long notice period). My team is pretty sad about it going away: https://cloud.google.com/debugger/docs

Using debugger for the first time was actually incredible. Having a debugger attached to a live service with zero impact was such an alien feeling. What a cool piece of tech.

Re: Google Cloud Europe service disruption

#98

Post by verdverm »

tinco wrote: post #56
rippercushions wrote: post #12

Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…

So this is probably too soon, thoughts and prayers for the datacenter operators and staff out there, but are they going to auction off the flooded hardware? Trying to restore a flooded Google rack sounds like a super fun project. Anyone experience with losing an entire DC to flooding? edit: I just Googled it (lol) and this DC has to be brand spanking new ( https://cloud.google.com/blog/products/infrastructure/google.…

I doubt they would let anyone have access to their hardware. There is a ton of proprietary stuff in there

Re: Google Cloud Europe service disruption

#99

Post by 8organicbits »

jsonne wrote: post #96
neeleshs wrote: post #79

Earlier quoted context omitted.

GCP doesn't operate the same way as Google consumer products. We are a paid customer for over 5 years and I also have only good things to say about GCP and their support

Really because I'm a GCP customer also and earlier this week they arbitrarily shut off Looker on us with no explanation leading to tons of pissed off customers. Our account executive responded with no help and a link to file a ticket. I expect a lot more from a service we're paying $10k+ a month for and my experience with Google has been so bad we're considering migrating everything to Microsoft.

Wait, what happened to Looker?

Re: Google Cloud Europe service disruption

#100

Post by Veliladon »

neeleshs wrote: post #79
tvanantwerp wrote: post #49

Earlier quoted context omitted.

This was my first thought too. Shows how Google has trained us to expect the worst from them...

GCP doesn't operate the same way as Google consumer products. We are a paid customer for over 5 years and I also have only good things to say about GCP and their support

>We are a paid customer

To be fair, so were Stadia users.

Post reply on HN