Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

131–140 of 341 posts

Re: Google Cloud networking issues in us-east1

#131

Post by lwb »

CaptainJustin wrote: post #101
mathattack wrote: post #91

Earlier quoted context omitted.

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

It's quite common in cloud solution design to design for failure. One of the common assumptions that we hold to is that one region may go down. Other examples: Assume an instance of an app can go down. Assume a VM can go down. Assume a DC can go down. This is not to excuse the downtime in any way.

Do people ever worry that an entire cloud provider may go down, or is that too unlikely of a case?

Re: Google Cloud networking issues in us-east1

#132

Post by [deleted] »

tscanausa wrote: post #87
thsowers wrote: post #3

Why so many problems at Google lately? Calendar down two weeks ago[0], and Google Cloud had a larger outage a month ago[1] [0]: https://news.ycombinator.com/item?id=20213092 [1]: https://news.ycombinator.com/item?id=20077421

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside…

[deleted]

Re: Google Cloud networking issues in us-east1

#134

Post by sudosteph »

user5994461 wrote: post #97
Johnny555 wrote: post #44

Earlier quoted context omitted.

AWS EC2 promises 4 9's (4.3 minutes of downtime/month) before their SLA kicks in, but they only give a 10% discount until availability dips below 99% (7.5 hours of downtime/month) when they give a 30% discount. If availability is below 95% (36 hours) in a month, they give a full refund. For an individual instance, they only promise 90% availability.

Availability of what? I've noticed entire afternoon where it wasn't possible to provision instances of some types, when I was working with AWS daily.

Availability of network access to existing instances.

What you're talking about with provisioning capacity is a totally different matter. Provisioning availability is not guaranteed (unless you purchase reserved instances) and there are frequently periods where certain instance types are not available in certain AZs, though they do try to resolve that as fast as practicality allows them to. It really stinks sometimes though - especially if you get into a situation where something fails in your autoscaling group and there is no capacity available for a replacement instance. Usually you can get around that though by making sure your ASG is set up for multiple AZs, or worst case changing instance types (though that can be problematic in it's own way).

source: I used to work for AWS Support.

Re: Google Cloud networking issues in us-east1

#135

Post by z3t4 »

When choosing a big cloud provider people forget that it's many orders of magnitude more complicated to run something at Google scale then to maintain one single server. For example the whole Stack overflow website runs on one or two servers. World of Warcraft also used to run on one single (blade) server. Chances are one server will be good enough for most use cases. And if you don't want to have it in your closet there are plenty of dedicated hosting and colocations.

Re: Google Cloud networking issues in us-east1

#136

Post by boulos »

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.).

As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad.

We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've removed almost all Google.com traffic out of the Region to prefer GCP customers. That's why the latency increase is subsiding, as we're freeing up the fiber paths by shedding our traffic.

Edit: (since it came up) that also means that if you’re using GCLB and have other healthy Regions, it will rebalance to avoid this congestion/slowdown automatically. That seemed the better trade off given the reduced network capacity during this outage.

[1] https://status.cloud.google.com/incident/cloud-networking/19...

Re: Google Cloud networking issues in us-east1

#137

Post by tlynchpin »

geogram wrote: post #108

Longer than 4 hours. We have stackdriver setup to monitor uptime/latency and its been acting up since 2am PST.

ObPedant: notice in google's status page "...as of Tuesday, 2019-07-02 09:11 US/Pacific." This notation is useful because it's stable year round. I don't recommend 'PDT', instead colloquially 'out here on the left coast' or specifically US/Pacific.

Re: Google Cloud networking issues in us-east1

#138

Post by geogram »

crankylinuxuser wrote: post #60
bob33212 wrote: post #2

Down or just high latency? For some folks that is the same thing.

is a 4 hr latency, "latency"? You make a good point though. Downtime seems to be awfully overloaded.

On our tests the latency is surprisingly low (20-40ms) but it has an error rate of 10-30%.

Re: Google Cloud networking issues in us-east1

#139

Post by ffk »

mathattack wrote: post #91
tscanausa wrote: post #87

Earlier quoted context omitted.

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside…

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

This is one of the reasons things like Federated Kubernetes is being worked on. Stick a CDN in front and your compute can be migrated from cloud to cloud. You still need to do a lot of thinking about data though.

Re: Google Cloud networking issues in us-east1

#140

Post by boulos »

fastest963 wrote: post #4

Here's the original issue: https://status.cloud.google.com/incident/cloud-networking/19... Not sure why they closed that one at 9:12 just to open a new one at 10:25. We didn't see any traffic coming to us-east1 during that time period so I would assume the original issue is still the root cause.

Yeah, that happens sometimes based on which team notices, thinks it might be different and then opens an outage.

Sorry for the confusion, and yes, the fiber link issue is the root cause. Draining the Google.com traffic presumably resolved the issue for you, though you may still be seeing elevated latency as the updates suggest.

Post reply on HN