I know I said our pipeline abstraction was leaky but this is ridiculous
Google Cloud region currently down due to water intrusion
61–70 of 187 posts
Re: Google Cloud region currently down due to water intrusion
#62eu-west-9 is Paris
Thanks for posting which one, this is the most important detail and should have been in the title...
Re: Google Cloud region currently down due to water intrusion
#63A perfect example of when "cloud just means someone else's computers". It's literally a leaky abstraction.
1) Pay for stuff
2) Not be able to use it
3) Company continues to pretend this doesn't happen on the regular
Re: Google Cloud region currently down due to water intrusion
#64[flagged]
Re: Google Cloud region currently down due to water intrusion
#65I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.
zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.
In the same building is quite a bit different than 10km apart, even if a huge meteor would lead to the same conclusion for both scenarios.
Re: Google Cloud region currently down due to water intrusion
#66I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.
zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.
Wow, this is a big issue! Is there any way to guarantee AWS-level physical redundancy in GCP without paying the latency/inter-region data transfer (higher than inter-zone outside US: https://cloud.google.com/vpc/network-pricing#egress-within-g...) pricing?
A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.
Re: Google Cloud region currently down due to water intrusion
#67Anyone know how this could affect multiple zones? "Customers can failover to zones in other regions". Unless a whole area got flooded.
There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
It amazes me that in every market they serve, Amazon has no actual competitors from a feature perspective.
Like, Target does not compete with Amazon. They have a totally different home delivery model that is not in the same category of reliability or service.
It's annoying.
Re: Google Cloud region currently down due to water intrusion
#68[flagged]
Re: Google Cloud region currently down due to water intrusion
#69Earlier quoted context omitted.
Lol us-east-1 already went down for a day back in 2017 when an intern accidentally took down the whole DC. We could call him “Godzilla” Source: my startup (stupidly) hosted our entire infra in us-east-1 at the time. Was a …tough day
us-east-1 is a great place to test your application resiliency :) it's like they threw in chaosmonkey for free!
It's funny because AWS, at least for the services I knew of when I was there, did rolling deploys to each region over several days. us-east-1 was always the final day because it was the biggest region, so you'd think it'd the safest region since everything getting deployed was well-tested. But while I was there I remember at least 2 COEs where the root cause was basically, "us-east-1 had some hacky legacy configuration that no other region has and that wasn't known/accounted for."
Re: Google Cloud region currently down due to water intrusion
#70A perfect example of when "cloud just means someone else's computers". It's literally a leaky abstraction.
1) Pay for stuff 2) Not be able to use it 3) Company continues to pretend this doesn't happen on the regular
> Company continues to pretend this doesn't happen on the regular
What do you want them to say? "Hey we have X breakdowns but please, pay!"