This is the problem storing data in the cloud - whenever it rains you may have a big data problem.
It's okay because once the water evaporates its backup in the cloud.
Google Cloud region currently down due to water intrusion
41–50 of 187 posts
Re: Google Cloud region currently down due to water intrusion
#42Earlier quoted context omitted.
You would think that the company that literally wrote the book on “Site Reliability Engineering” would actually follow their own recommendations.
Do Google host their own products on Google Cloud, or are there different sets of data centres for Search/Drive/Gmail vs Google Cloud Customers?
This is a leased facility, the kind of place Google rents for cloud customers but doesn't need for itself. Google's own datacenters are https://www.google.com/about/datacenters/locations/
Re: Google Cloud region currently down due to water intrusion
#43Earlier quoted context omitted.
You would think that the company that literally wrote the book on “Site Reliability Engineering” would actually follow their own recommendations.
Do Google host their own products on Google Cloud, or are there different sets of data centres for Search/Drive/Gmail vs Google Cloud Customers?
Other way around. Google Cloud runs on the same underlying datacenter, compute, and network infrastructure that Search/Drive/Gmail does.
[edit: at least in regions where Google HAS its own datacenters, e.g. "us-central-1? yes. europe-west9? maybe not"].
That does not imply that Search / Drive / Gmail runs on top of Google Cloud.
Re: Google Cloud region currently down due to water intrusion
#44Earlier quoted context omitted.
The headline is not incorrect. "Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be…
> As of a few hours ago they changed the status to report just on us-west9-a. The headline is 1h20m old and says a region is currently down.
Even the article linked mentions that Google initially reported that only zone A was affected, then got changed to report that the whole region was affected, now it changed to a single zone again.
Do you expect realtime updates whenever Google changes the story?
Re: Google Cloud region currently down due to water intrusion
#45Anyone know how this could affect multiple zones? "Customers can failover to zones in other regions". Unless a whole area got flooded.
There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
I would really like to see the physical DC separation at "The Dalles, Oregon".
Re: Google Cloud region currently down due to water intrusion
#46I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.
Re: Google Cloud region currently down due to water intrusion
#47A series of tubes would have helped with this.
Thank you for your input, Senator Stevens.
Re: Google Cloud region currently down due to water intrusion
#48Earlier quoted context omitted.
There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
For physical zone separation you need to check the `supportsPzs` attribute when listing the zones (e.g. https://cloud.google.com/compute/docs/reference/rest/v1/zone... , but you should be able to find many other places where this attribute is surfaced). It says "reserved for future use" but other docs mentioned "physical zone separation": https://googleapis.dev/java/google-api-services-compute/alph...
Random datacenters should start advertising availability zones since they should have different fault domains anyway. Google can get away with this, why can't smaller companies?
Re: Google Cloud region currently down due to water intrusion
#49This seems to significantly under-report what's going on, see: https://www.theregister.com/2023/04/26/google_cloud_outage/ There is mention of a fire as well.
Plot twist: the server racks were made out of sodium.
Re: Google Cloud region currently down due to water intrusion
#50I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.
zones and regions have different definition in google cloud than AWS.
Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.