Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

191–200 of 341 posts

Re: Google Cloud networking issues in us-east1

#191

Post by dragonwriter »

mathattack wrote: post #91
tscanausa wrote: post #87

Earlier quoted context omitted.

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside…

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

More than one region is pretty easy, more than one provider is harder (especially if your workload is designed from the ground up for it.) But, yes, just as multi-region protects you from things mere multi-AZ doesn't, multi-provider protects you from even more.

Re: Google Cloud networking issues in us-east1

#192

Post by dragonwriter »

mbesto wrote: post #145
benbro wrote: post #98

Earlier quoted context omitted.

Is it practical to use several providers when egress is so expensive?

No. And there's been a lot of talk recently about multi-provider being the right strategy to mitigate downtime, which IMHO is a farce peddled by expensive consultants. The parent comment is correct - this is why availability zones and regions have been established by each provider. For the large majority of businesses investing in infrastructure-as-code far outweighs any crazy HA, redundant, multi-provider, whizzbang…

> this is why availability zones and regions have been established by each provider.

But the degree of independence provided by AZs is not constant across providers, despite similar terminology.

Re: Google Cloud networking issues in us-east1

#193

Post by ricardobeat »

boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

Re: Google Cloud networking issues in us-east1

#194

Post by orf »

samcday wrote: post #147
alteria wrote: post #126

Earlier quoted context omitted.

I'm curious - what are some examples of the warts you encounter?

Things like: * filtering traces by services has been broken in App Engine flex environments for more than a year. * copy/pasting identifiers between places is a nightmare * their IAM design is somehow worse than AWS. It’s so impressively bad I can’t even be mad. My favourite part of their IAM approach is how they have consolidated a majority of the IAM controls in the IAM page, but then random services like GCS have…

> * multi-account paper cuts

I always considered the Google Cloud approach of a "single account, multiple projects" a lot cleaner than the AWS "hundreds of accounts" approach. Do you not find this the case?

Re: Google Cloud networking issues in us-east1

#195

Post by dragonwriter »

inlined wrote: post #6

Holy crap. It’s an outage in all zones? What’s the point of AZs if you lose whole DCs at a time.

> What’s the point of AZs if you lose whole DCs at a time.

The point is that AZs are higher level than DCs, so that they provide pretty decent independence guarantees (though you can further derisk with multi-region.)

Well, in AWS. Google's zones have weaker independence assurances (actually, as I read it, no assurances), stating only that a zone “usually has power, cooling, networking, and control planes that are isolated from other zones” [0] as opposed to AWS’s “Availability Zones are physically separated within a typical metropolitan region” and “In addition to discrete uninterruptable power supply (UPS) and onsite backup generation facilities, they are each fed via different grids from independent utilities to further reduce single points of failure. Availability Zones are all redundantly connected to multiple tier-1 transit providers.” [1]

[0] https://cloud.google.com/compute/docs/regions-zones/

[1] https://docs.aws.amazon.com/whitepapers/latest/aws-overview/...

Re: Google Cloud networking issues in us-east1

#196

Post by jauer »

ricardobeat wrote: post #193
boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

It's probably less "as they wish" and more "here's an approved statement" or "your role involves engaging with external parties, here are some guidelines"

Re: Google Cloud networking issues in us-east1

#197

Post by kyrra »

ricardobeat wrote: post #193
boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

It's a fine line. We are not allowed to represent Google in any kind of public discussion. But we can talk about some things we do, as long as we state it's our own opinion and we don't represent Google's views.

Re: Google Cloud networking issues in us-east1

#198

Post by edwintorok »

boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

You seem to have 3 status messages on the dashboard at 14:31, 14:44 and 14:48 with exactly the same contents. Were those messages really posted 3 times, or did something go wrong and they got duplicated?

Re: Google Cloud networking issues in us-east1

#199

Post by davedunkin »

avocado4 wrote: post #184
z3t4 wrote: post #135

When choosing a big cloud provider people forget that it's many orders of magnitude more complicated to run something at Google scale then to maintain one single server. For example the whole Stack overflow website runs on one or two servers. World of Warcraft also used to run on one single (blade) server. Chances are one server will be good enough for most use cases. And if you don't want to have it in your closet t…

How can Stack Overflow run on a single server? Do you mean single cluster?

As of 2016, Stack Overflow ran on dozens of servers in two data centers.

https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...

Re: Google Cloud networking issues in us-east1

#200

Post by edwintorok »

z3t4 wrote: post #135

When choosing a big cloud provider people forget that it's many orders of magnitude more complicated to run something at Google scale then to maintain one single server. For example the whole Stack overflow website runs on one or two servers. World of Warcraft also used to run on one single (blade) server. Chances are one server will be good enough for most use cases. And if you don't want to have it in your closet t…

With a cloud it also means that when there is an outage there are potentially many sites/services affected all at once, and there is potentially nothing customers can do to fix it other than wait (or plan in advance, and use/pay for multi-AZ/multi-region/multi-provider redundancy). Such outages are also possible with traditional hosting providers, and when an outage does happen I'm not convinced whether a large public cloud would recover more quickly (due to better resourcing/expertise available to fix the problem), or a small hosting provider (which may have a smaller team, but the problems they deal with are at a smaller scale and more easily fixable). Either way you probably want some kind of CDN independent of your cloud/hosting provider that can help survive some of these glitches.
Post reply on HN