Google cloud outage

#1

Post by thomassharoon » Fri, Mar 27, 2020, 8:42 PM UTC

Google cloud outage
status.cloud.google.com

Re: Google cloud outage

#2

Post by nammi » Fri, Mar 27, 2020, 8:47 PM UTC

We were seeing timeouts in east-1. I don't know what "normal" looks like, but Pingdom's map seems to show the whole east coast as affected https://livemap.pingdom.com/

Re: Google cloud outage

#3

Post by svacko » Fri, Mar 27, 2020, 9:16 PM UTC

yeah, our GKE pods running in us-east1 were dying ~90minutes ago like crazy... hope they are gonna resolve this soon. not the luckiest day for Google, nor us

Re: Google cloud outage

#4

Post by qmarchi » Fri, Mar 27, 2020, 9:18 PM UTC

Heyo Googler here.

The problem was a mix between another cloud provider and GCP.

Dare I say, there should be little customer impact as of 13:37 PST.....

The status dashboard is going to be your best idea on information.

Re: Google cloud outage

#5

Post by the-dude » Fri, Mar 27, 2020, 9:24 PM UTC

qmarchi wrote: post #4

Heyo Googler here. The problem was a mix between another cloud provider and GCP. Dare I say, there should be little customer impact as of 13:37 PST..... The status dashboard is going to be your best idea on information.

This can't be real.

Re: Google cloud outage

#6

Post by svacko » Fri, Mar 27, 2020, 9:27 PM UTC

qmarchi wrote: post #4

Heyo Googler here. The problem was a mix between another cloud provider and GCP. Dare I say, there should be little customer impact as of 13:37 PST..... The status dashboard is going to be your best idea on information.

Is the another cloud provider AWS? I could see tons of connection timeoutes between GCP & S3/Elasticsearch service.

Hope everything is resolved now for good.

Re: Google cloud outage

#7

Post by judge2020 » Fri, Mar 27, 2020, 9:30 PM UTC

svacko wrote: post #6

qmarchi wrote: post #4

Heyo Googler here. The problem was a mix between another cloud provider and GCP. Dare I say, there should be little customer impact as of 13:37 PST..... The status dashboard is going to be your best idea on information.

Is the another cloud provider AWS? I could see tons of connection timeoutes between GCP & S3/Elasticsearch service. Hope everything is resolved now for good.

Seems AWS, connection to gmail's smtp relay also started timing out.

Re: Google cloud outage

#8

Post by [deleted] » Fri, Mar 27, 2020, 9:31 PM UTC

svacko wrote: post #6

qmarchi wrote: post #4

Heyo Googler here. The problem was a mix between another cloud provider and GCP. Dare I say, there should be little customer impact as of 13:37 PST..... The status dashboard is going to be your best idea on information.

Is the another cloud provider AWS? I could see tons of connection timeoutes between GCP & S3/Elasticsearch service. Hope everything is resolved now for good.

[deleted]

Re: Google cloud outage

#9

Post by tagux » Fri, Mar 27, 2020, 9:37 PM UTC

"We had a router failure in Atlanta".

WHAT? You kidding us?

Urs Hölzle, technical infrastructure at Google Cloud senior vice president, said, "We're very sorry about that! We had a router failure in Atlanta, which affected traffic routed through that region. Things should be back to normal now. Just to make sure: This wasn't related to traffic levels or any kind of overload, our network is not stressed by COVID-19."

Re: Google cloud outage

#10

Post by ocdtrekkie » Fri, Mar 27, 2020, 9:40 PM UTC

tagux wrote: post #9

"We had a router failure in Atlanta". WHAT? You kidding us? Urs Hölzle, technical infrastructure at Google Cloud senior vice president, said, "We're very sorry about that! We had a router failure in Atlanta, which affected traffic routed through that region. Things should be back to normal now. Just to make sure: This wasn't related to traffic levels or any kind of overload, our network is not stressed by COVID-19."

Was it like... a hardware failure? If you serve more than 100 people you probably should have redundant routers. Was it a configuration issue that replicated over to multiple devices at least, I hope?