Live data from Hacker News

Google cloud outage

status.cloud.google.com

21–30 of 39 posts

Re: Google cloud outage

#21
post #16

Earlier quoted context omitted.

[removed]

Next time YOU are about to spout off about something, perhaps think about reading the f'ing page being linked to? "The issue with connectivity between the GCP us-east1, us-east4, and us-central1 regions to other Cloud Providers has been resolved for all affected projects as of Friday, 2020-03-27 13:37 US/Pacific."

Perhaps you're right. I'll better discuss it privately.

Re: Google cloud outage

#22

Earlier quoted context omitted.

Oh man I had no idea the big cloud providers have dependencies on other clouds like this.

Given how much trans-continental/trans-oceanic network cable the major cloud providers own, they almost certainly have special trans-cloud network traffic infrastructure. Especially since so much of "The Cloud" is within a few 10s of square miles in a field in Virginia. I can easily see how one provider could majorly disrupt another provider by accidentally breaking inbound traffic on one of those links.

The bigger issue is that there's a lot of customers where they have split cloud deployments, which means the customers hurt even if they are stable within the clouds themselves.

Re: Google cloud outage

#23
post #15

Earlier quoted context omitted.

Was it like... a hardware failure? If you serve more than 100 people you probably should have redundant routers. Was it a configuration issue that replicated over to multiple devices at least, I hope?

Have you worked with redundant routers? They certainly reduce the number of outages, but sometimes the hardware (or software) fails in exciting ways that doesn't engage the redundancy, or doesn't engage it properly, and you still get an outage (or you get an outage that wouldn't have happened). Or sometimes, one circuit is out of service for repair or upgrade, and the other circuit is connected to the router that fai…

I have. I am just highlighting that the problem surely should be more complex than described. Or that their redundancy for these events was not adequately devised.

Re: Google cloud outage

#24
post #4

Heyo Googler here. The problem was a mix between another cloud provider and GCP. Dare I say, there should be little customer impact as of 13:37 PST..... The status dashboard is going to be your best idea on information.

[deleted]

Re: Google cloud outage

#25

Earlier quoted context omitted.

Oh man I had no idea the big cloud providers have dependencies on other clouds like this.

Given how much trans-continental/trans-oceanic network cable the major cloud providers own, they almost certainly have special trans-cloud network traffic infrastructure. Especially since so much of "The Cloud" is within a few 10s of square miles in a field in Virginia. I can easily see how one provider could majorly disrupt another provider by accidentally breaking inbound traffic on one of those links.

Yeah, I see that now. Makes total sense.

Re: Google cloud outage

#26
post #4

Heyo Googler here. The problem was a mix between another cloud provider and GCP. Dare I say, there should be little customer impact as of 13:37 PST..... The status dashboard is going to be your best idea on information.

Oh man I had no idea the big cloud providers have dependencies on other clouds like this.

They do not, according to the dashboard, this issue merely affected connectivity between GCP and other cloud providers.

There was a different outage yesterday, which has nothing to do with the one discussed in this thread.

Re: Google cloud outage

#27
post #22

Earlier quoted context omitted.

Given how much trans-continental/trans-oceanic network cable the major cloud providers own, they almost certainly have special trans-cloud network traffic infrastructure. Especially since so much of "The Cloud" is within a few 10s of square miles in a field in Virginia. I can easily see how one provider could majorly disrupt another provider by accidentally breaking inbound traffic on one of those links.

The bigger issue is that there's a lot of customers where they have split cloud deployments, which means the customers hurt even if they are stable within the clouds themselves.

If you are deployed in such a way that both GCP and AWS need to be up you're doing it backwards. Multi-cloud strategy is supposed to result in the intersection of cloud failures, not the union of them.

Re: Google cloud outage

#28
post #9

"We had a router failure in Atlanta". WHAT? You kidding us? Urs Hölzle, technical infrastructure at Google Cloud senior vice president, said, "We're very sorry about that! We had a router failure in Atlanta, which affected traffic routed through that region. Things should be back to normal now. Just to make sure: This wasn't related to traffic levels or any kind of overload, our network is not stressed by COVID-19."

Wrong outage.

Re: Google cloud outage

#29
post #22

Earlier quoted context omitted.

The bigger issue is that there's a lot of customers where they have split cloud deployments, which means the customers hurt even if they are stable within the clouds themselves.

If you are deployed in such a way that both GCP and AWS need to be up you're doing it backwards. Multi-cloud strategy is supposed to result in the intersection of cloud failures, not the union of them.

"But all of our problems are fixed by going to the cloud!"

Re: Google cloud outage

#30
post #16

Earlier quoted context omitted.

[removed]

Next time YOU are about to spout off about something, perhaps think about reading the f'ing page being linked to? "The issue with connectivity between the GCP us-east1, us-east4, and us-central1 regions to other Cloud Providers has been resolved for all affected projects as of Friday, 2020-03-27 13:37 US/Pacific."

> Next time YOU are about to spout off about something, perhaps think about reading the f'ing page being linked to?

this is very cruel, there is absolutely no need to be so harsh.

Post reply on HN