Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ongoing Incident in Google Cloud

status.cloud.google.com

Post reply on HN
Print view View original

51–60 of 115 posts

Re: Ongoing Incident in Google Cloud

#51

Post by zamnos »

bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

The underlying problem is that Google doesn't operate the world's DNS servers, but still wants to offer the best possible user experience as a global service. This means anycast VIP routing, because not all DNS servers implement EDNS, but they want to have SSL connections terminate as closely to users as possible.

As far as global services go though, it's easy enough to say "it should just not be possible", but how do you propose doing that in practice for a global service?

How does new config going to go out, globally, without being global? How do global services work if they're not global? How does DDoS protection work if you don't do it globally?

People make fun of "webscale" but operating Google is really difficult and complicated!

Re: Ongoing Incident in Google Cloud

#52

Post by joatmon-snoo »

throwaway892238 wrote: post #34
uniformlyrandom wrote: post #28

05:41 - 06:26 PT, 45 min total. Not great, not terrible.

Yep. Of course there's no detail yet so we don't know what exactly was affected. All we can see is "Multiple services are being impacted globally" and a list of services (Build, Firestore, Container Registry, BigQuery, Bigtable, Networking, Pub/Sub, Storage, Compute Engine, Identity and Access Management) but there's no indication of what specifically was impacted. Could you still see status for your VMs, but not lau…

Ex-GCP here.

This is a pretty reductionist summary, e.g. the 8-day Cloud Networking incident root cause:

> Description: Our engineering team continues to investigate this issue and is evaluating additional improvement opportunities to identify effective rerouting of traffic. They have narrowed down the issue to one regional telecom service provider and reported this to them for further investigation. The connectivity problems are still mostly resolved at this point although some customers may observe delayed round trip time or longer latency or sporadic packet loss until fully resolved.

Still a big problem product-wise, but you're looking at a global incident history view without any region/severity filters.

The corresponding AWS service health dashboard makes it much harder to view this level of detail, but is also actually useful for someone asking "is product $xyz which I depend on in region $abc currently down or not"

Re: Ongoing Incident in Google Cloud

#53

Post by dixie_land »

fastest963 wrote: post #3

This affected us starting at 4:57am US/Pacific with a significant drop in traffic through the HTTPS Global Load Balancer across all regions and Pub/Sub 502 errors but there was nothing on the status page for another 45 minutes. Things returned to normal by 5:05am from what I can tell.

Yup we saw the exact same symptoms with some GCLBs getting 100% 502 ( our upstream QPS graph looks scary with 5 mins of 0 QPS )

Re: Ongoing Incident in Google Cloud

#54

Post by otterley »

zamnos wrote: post #51
bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

The underlying problem is that Google doesn't operate the world's DNS servers, but still wants to offer the best possible user experience as a global service. This means anycast VIP routing, because not all DNS servers implement EDNS, but they want to have SSL connections terminate as closely to users as possible. As far as global services go though, it's easy enough to say "it should just not be possible", but how d…

https://aws.amazon.com/builders-library/automating-safe-hand...

Re: Ongoing Incident in Google Cloud

#55

Post by zamnos »

consumer451 wrote: post #11
bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

My knowledge level: can use AWS console to do How much more work would Google create for themselves if they had not globalized their stack? Are we talking something like 5 subsets to manage instead of 1?

If you get into the nitty gritty of it, it doesn't really make sense. Are you going to have 5 different load balancer software stacks, with 5 different config file languages, causing each client (say Gmail) to have to implement their config 5 different ways? That's insane.

Re: Ongoing Incident in Google Cloud

#56

Post by crazygringo »

monero-xmr wrote: post #31

This is why any criticism of AWS reliability is meaningless to me. All the cloud providers go down - all of them. Either you are multi-cloud, or you run your own hardware, but these events are inevitable.

> Either you are multi-cloud, or you run your own hardware

If you run your own hardware these events are inevitable too.

Re: Ongoing Incident in Google Cloud

#57

Post by kevinventullo »

bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

I reckon the only to achieve that would be to have the same level of interoperability between regions as you would get between two distinct cloud providers.

Re: Ongoing Incident in Google Cloud

#58

Post by andsoitis »

zamnos wrote: post #41
andsoitis wrote: post #15

Earlier quoted context omitted.

Solving this sort of thing is not about throwing more people at it. That would be brute force and not strategic. Instead, you want to architect systems like these in a way that strikes a good balance between resilience and things like cost/efficiency/etc.

Sure, but laying off a number of good SREs from the Internet traffic team that is responsible for the Google Load Balancer team can't be helping this situation.

How many such people were laid off and what were their unique expertise?

Re: Ongoing Incident in Google Cloud

#59

Post by zamnos »

otterley wrote: post #54
zamnos wrote: post #51

Earlier quoted context omitted.

The underlying problem is that Google doesn't operate the world's DNS servers, but still wants to offer the best possible user experience as a global service. This means anycast VIP routing, because not all DNS servers implement EDNS, but they want to have SSL connections terminate as closely to users as possible. As far as global services go though, it's easy enough to say "it should just not be possible", but how d…

https://aws.amazon.com/builders-library/automating-safe-hand...

AWS US east 1 had significant downtime last year so I'm not sure what you're trying to say with that link. Would you mind expanding on your thoughts?
Post reply on HN