Ongoing Incident in Google Cloud

#1

Post by sd2k » Mon, Feb 27, 2023, 1:47 PM UTC

Ongoing Incident in Google Cloud
status.cloud.google.com

Re: Ongoing Incident in Google Cloud

#2

Post by lokl » Mon, Feb 27, 2023, 2:10 PM UTC

mail.google.com showed error messages for me intermittently during the past hour.

Re: Ongoing Incident in Google Cloud

#3

Post by fastest963 » Mon, Feb 27, 2023, 2:13 PM UTC

This affected us starting at 4:57am US/Pacific with a significant drop in traffic through the HTTPS Global Load Balancer across all regions and Pub/Sub 502 errors but there was nothing on the status page for another 45 minutes. Things returned to normal by 5:05am from what I can tell.

Re: Ongoing Incident in Google Cloud

#4

Post by asicsp » Mon, Feb 27, 2023, 2:16 PM UTC

lokl wrote: post #2

mail.google.com showed error messages for me intermittently during the past hour.

discussed here: https://news.ycombinator.com/item?id=34955906

Re: Ongoing Incident in Google Cloud

#5

Post by jakedata » Mon, Feb 27, 2023, 2:17 PM UTC

lokl wrote: post #2

mail.google.com showed error messages for me intermittently during the past hour.

https://www.google.com/appsstatus/dashboard/incidents/5ML14k...

They claim the Gmail specific issues are resolved. We shall see...

Feb 27, 2023 2:03 PM UTC We experienced a brief network outage with package loss, impacting a number of workspace services. The impact is over. We are investigating and monitoring.

Re: Ongoing Incident in Google Cloud

#6

Post by 0x0000000 » Mon, Feb 27, 2023, 2:18 PM UTC

Outages at the hyperscalers can have a huge blast radius, is anyone encountering other services with outages because they're built on GCP?

Re: Ongoing Incident in Google Cloud

#7

Post by pictur » Mon, Feb 27, 2023, 2:20 PM UTC

[flagged]

Re: Ongoing Incident in Google Cloud

#8

Post by bushbaba » Mon, Feb 27, 2023, 2:20 PM UTC

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern.

gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

Re: Ongoing Incident in Google Cloud

#9

Post by typaty » Mon, Feb 27, 2023, 2:31 PM UTC

https://packages.cloud.google.com/apt/doc/apt-key.gpg Even the public apt key for signing Google's cloud packages is unavailable (returns 500 for me). This is insane

Re: Ongoing Incident in Google Cloud

#10

Post by nicholasklem » Mon, Feb 27, 2023, 2:35 PM UTC

typaty wrote: post #9

https://packages.cloud.google.com/apt/doc/apt-key.gpg Even the public apt key for signing Google's cloud packages is unavailable (returns 500 for me). This is insane

This key was 500 some hours before the incident started, I hope it's unrelated.