Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud region currently down due to water intrusion

status.cloud.google.com

Post reply on HN
Print view View original

81–90 of 187 posts

Re: Google Cloud region currently down due to water intrusion

#82

Post by ec109685 »

t0mas88 wrote: post #46

I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.

It was not only a region outage, but a global outage!

"GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating... "

Embarrassing.

Re: Google Cloud region currently down due to water intrusion

#84

Post by londons_explore »

migf wrote: post #67
gst wrote: post #5

Earlier quoted context omitted.

There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.

It amazes me that in every market they serve, Amazon has no actual competitors from a feature perspective. Like, Target does not compete with Amazon. They have a totally different home delivery model that is not in the same category of reliability or service. It's annoying.

I think it's because lots of amazons services are in 'winner takes all' markets.

No random online eshop can offer next day delivery across half the world unless they already have a logistics chain of 100,000 truck drivers spread across the world. But Amazon can.

Likewise, no cloud provider has enough data centers to offer multiple separate data centers in the same city, for hundreds of cities around the world. But Amazon does.

Any competitor can't offer amazons level of service until they get to amazon scale... Which they never will.

Re: Google Cloud region currently down due to water intrusion

#85

Post by okdood64 »

lamontcg wrote: post #73

Nobody here with any thoughts for the operations/datacenter engineers trying to deal with stopping and cleaning up the disaster, just customers complaining...

Nope. Big company bad.

Literally no concern here for anyone's safety or sanity in dealing with this.

Re: Google Cloud region currently down due to water intrusion

#88

Post by londons_explore »

xyst wrote: post #17
gst wrote: post #5

Earlier quoted context omitted.

There's an interesting Twitter thread about that topic here: https://twitter.com/GergelyOrosz/status/1651256082424012806 Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.

You would think that the company that literally wrote the book on “Site Reliability Engineering” would actually follow their own recommendations.

Googles advice is not to rely on uptime in every region.

Instead aim for uptime in a few regions, and load balance your users to regions that are healthy.

That design is far cheaper for both google and for you - and, in the typical case, users still get nice low latency to a local datacenter, and only in the rare failure case might they have to wait for latency to some other region.

Post reply on HN