A series of tubes would have helped with this.
Google Cloud region currently down due to water intrusion
31–40 of 187 posts
Re: Google Cloud region currently down due to water intrusion
#32Earlier quoted context omitted.
There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
You would think that the company that literally wrote the book on “Site Reliability Engineering” would actually follow their own recommendations.
The recommendations are to run in multiple regions if you need this kind of redundancy. Run everything in a single region and you can be affected by an event like this.
Re: Google Cloud region currently down due to water intrusion
#33[flagged]
> Due to software bugs other zones in the same region were also previously down.
I don't think it was a software bug. I think they were taken down as a precautionary measure, to not risk flood damage or causing additional fires.
Re: Google Cloud region currently down due to water intrusion
#34Earlier quoted context omitted.
There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
Yes. Azure and GCPs numbers on the size of their AZs and such are more marketing spin than hard engineering. AWS keeps these in separate physical locations to provide true separation. While there have been tech related regional incidents at AWS a physical event disabling multiple AZs would be extremely unlikely given their much more robust and geographically distributed design. If such a physical event had happened i…
> If such a physical event had happened in AWS it would have been a non-event with things just failing over to other AZs.
In many cases in AWS an availability zone is actually composed of multiple datacenters, each with their own redundancies. This may not be true for smaller regions, but in large ones it definitely is. In those cases, losing an entire datacenter would maybe take out a percentage of instances in that AZ. This has happened before and our production systems barely noticed other than provisioning new nodes to replace the failed health checks.
Re: Google Cloud region currently down due to water intrusion
#35Earlier quoted context omitted.
There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
AWS treats its availability zones very seriously, each zone has its own independent power substation, air conditioning, and fiber lines. It's incredibly rare for multiple AZs to go down at once, especially since they are more than a few miles apart from each other.
And different flood planes. Source: I was at AWS 2008-2014.
Re: Google Cloud region currently down due to water intrusion
#36Google now has a "data lake" in Paris.
I hope whoever is hosting data in that zone has thoroughly tested and verified backups offline or with another cloud provider. Of course, you can complete DC failover, depending upon service needs, but it costs more resources. Either way, timely tested backups are the only way to survive natural or manufactured disasters. Good luck to Google OPs team and everyone else involved with the GCP region in the EU.
Re: Google Cloud region currently down due to water intrusion
#37Anyone know how this could affect multiple zones? "Customers can failover to zones in other regions". Unless a whole area got flooded.
There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
For physical zone separation you need to check the `supportsPzs` attribute when listing the zones (e.g. https://cloud.google.com/compute/docs/reference/rest/v1/zone..., but you should be able to find many other places where this attribute is surfaced).
It says "reserved for future use" but other docs mentioned "physical zone separation": https://googleapis.dev/java/google-api-services-compute/alph...
Re: Google Cloud region currently down due to water intrusion
#38[flagged]
The headline is not incorrect.
"Multiple Google Cloud services in the europe-west9 region are impacted.
Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage"
Emergency shutdown of multiple zones.
As of a few hours ago they changed the status to report just on us-west9-a.
Re: Google Cloud region currently down due to water intrusion
#39[flagged]
The headline is not incorrect. "Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be…
> As of a few hours ago they changed the status to report just on us-west9-a.
The headline is 1h20m old and says a region is currently down.
Re: Google Cloud region currently down due to water intrusion
#40[flagged]
> Due to software bugs other zones in the same region were also previously down. I don't think it was a software bug. I think they were taken down as a precautionary measure, to not risk flood damage or causing additional fires.
and you're basing that on... what, exactly?