Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ongoing Incident in Google Cloud

status.cloud.google.com

Post reply on HN
Print view View original

11–20 of 115 posts

Re: Ongoing Incident in Google Cloud

#11

Post by consumer451 »

bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

My knowledge level: can use AWS console to do How much more work would Google create for themselves if they had not globalized their stack? Are we talking something like 5 subsets to manage instead of 1?

Re: Ongoing Incident in Google Cloud

#12

Post by Cthulhu_ »

bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

For reference / comparison, how many regional outages have there been? Did service outages get avoided due to running a workload in multiple regions?

Re: Ongoing Incident in Google Cloud

#13

Post by typaty »

nicholasklem wrote: post #10
typaty wrote: post #9

https://packages.cloud.google.com/apt/doc/apt-key.gpg Even the public apt key for signing Google's cloud packages is unavailable (returns 500 for me). This is insane

This key was 500 some hours before the incident started, I hope it's unrelated.

inb4 it turns out an intern was tasked with updating the apt key, which brought a cascading outage of all their services

Re: Ongoing Incident in Google Cloud

#15

Post by andsoitis »

pictur wrote: post #7

[flagged]

Solving this sort of thing is not about throwing more people at it. That would be brute force and not strategic. Instead, you want to architect systems like these in a way that strikes a good balance between resilience and things like cost/efficiency/etc.

Re: Ongoing Incident in Google Cloud

#16

Post by roseway4 »

typaty wrote: post #9

https://packages.cloud.google.com/apt/doc/apt-key.gpg Even the public apt key for signing Google's cloud packages is unavailable (returns 500 for me). This is insane

Downloading the key has been erroring since at least ~5pm PT yesterday, 2/27. It’s likely unrelated. Though I’d be unsurprised if the recent layoffs contributed to the situation.

Re: Ongoing Incident in Google Cloud

#17

Post by NineStarPoint »

consumer451 wrote: post #11
bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

My knowledge level: can use AWS console to do How much more work would Google create for themselves if they had not globalized their stack? Are we talking something like 5 subsets to manage instead of 1?

Assuming good automation, most of the work comes in being able to do a second of something instead of just having one. The difference in work between “single point” and “multiple point” is a lot, but increasing the multiple points beyond that isn’t too bad.

Of course, if you deploy a change to all of your separated stacks at once through some sort of automated pipeline it doesn’t matter too much. Easy to break everything simultaneously that way if there’s some difference between test and prod you didn’t realize was there.

Re: Ongoing Incident in Google Cloud

#18

Post by jjoonathan »

consumer451 wrote: post #11
bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

My knowledge level: can use AWS console to do How much more work would Google create for themselves if they had not globalized their stack? Are we talking something like 5 subsets to manage instead of 1?

My biggest AWS surprise bill (so far!) was due to a bug in AWS console region switching.

Re: Ongoing Incident in Google Cloud

#19

Post by medler »

bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

> gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

As I understand it, GCP is already designed to make global outages impossible. Obviously this outage shows that they messed up somehow and some global point of failure still remains. Looking forward to the post-mortem.

Re: Ongoing Incident in Google Cloud

#20

Post by singron »

consumer451 wrote: post #11
bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

My knowledge level: can use AWS console to do How much more work would Google create for themselves if they had not globalized their stack? Are we talking something like 5 subsets to manage instead of 1?

Most of it is cellular or regional, but there are a few critical global services. The global network load balancing, network qos, and ddos prevention are more functional because they are global (i.e. you couldn't replace them with equivalent regional versions), but are often causes of issues like this. There was a push a few years ago to ensure global services had at least 99.999% uptime or make them regional. This was a 48 minute outage, so it blows that five 9 budget for 9 years.

Ex-googler, no particular knowledge of this event, information might be out of date.

Post reply on HN