Google Cloud networking issues in us-east1 • Hacker News Board

Re: Google Cloud networking issues in us-east1

#221

Post by m0zg » Tue, Jul 02, 2019, 11:34 PM UTC

Operyl wrote: post #218

m0zg wrote: post #215

Earlier quoted context omitted.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

My horribly out of date system works, therefore I should never strive to improve it or god forbid update it (since that involves “fucking with it” in ways that can break it from version to version)? That gets you technical debt and that’s not fun.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operating systems underneath it all. Don't screw with it and it will work for a decade. Start screwing with it and it will be busted every month, like Google Cloud.

You can't create "technical debt" if you don't change anything in the first place.

Re: Google Cloud networking issues in us-east1

#222

Post by badrequest » Tue, Jul 02, 2019, 11:37 PM UTC

boulos wrote: post #202

ricardobeat wrote: post #193

Earlier quoted context omitted.

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

As kyrra says below, you're in the clear if you state that this is just your opinion. Naturally, prefacing something terrible as "just your opinion" doesn't make it fine. In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opin…

thank you very much for your candor!

Re: Google Cloud networking issues in us-east1

#223

Post by hunter2_ » Tue, Jul 02, 2019, 11:40 PM UTC

digitalsanctum wrote: post #217

I routinely see notices of outages like this posted on HN while HN itself never seems to be impacted. This begs the question: Where and how is HN hosted in a way that avoids being impacted by widespread network and provider outages?

  $ host news.ycombinator.com
  news.ycombinator.com has address 209.216.230.240

https://whois.arin.net/rest/net/NET-209-216-230-0-1/pft?s=20...

M5 Computer Security

https://www.m5hosting.com

Unrelated: https://begthequestion.info/

Re: Google Cloud networking issues in us-east1

#224

Post by jsjohnst » Tue, Jul 02, 2019, 11:48 PM UTC

digitalsanctum wrote: post #217

I routinely see notices of outages like this posted on HN while HN itself never seems to be impacted. This begs the question: Where and how is HN hosted in a way that avoids being impacted by widespread network and provider outages?

I’ve seen a few outages that impacted HN from a provider standpoint. Good example is the CenturyLink outage a few weeks back. CenturyLink isn’t my ISP and neither is it directly HN’s either, but my route to HN was impacted by the outage.

Re: Google Cloud networking issues in us-east1

#225

Post by Terretta » Tue, Jul 02, 2019, 11:49 PM UTC

Terretta wrote: post #59

aNoob7000 wrote: post #27

Earlier quoted context omitted.

No, He's just talking about all the people running around right now trying to figure out what went wrong and how to fix it. Criticizing Google is fine, but sometimes, the best deployments to production can go wrong.

If there’s a different deployment that could have worked, then the one that did go wasn’t the “best”. Critique should be around the 5 whys the actual best wasn’t selected for.

Turns out this is the conclusion in CloudFlare’s update:

> “Our testing processes were insufficient in this case and we are reviewing and making changes to our testing and deployment process to avoid incidents like this in the future.”

They’re going after the definition of good for their deployments.

Re: Google Cloud networking issues in us-east1

#226

Post by cortesoft » Tue, Jul 02, 2019, 11:49 PM UTC

Operyl wrote: post #218

m0zg wrote: post #215

Earlier quoted context omitted.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

My horribly out of date system works, therefore I should never strive to improve it or god forbid update it (since that involves “fucking with it” in ways that can break it from version to version)? That gets you technical debt and that’s not fun.

Technical debt is about the increasing difficulty to add features to a system.... if you aren't adding features, technical debt is not really an issue.

Re: Google Cloud networking issues in us-east1

#227

Post by munificent » Tue, Jul 02, 2019, 11:54 PM UTC

ricardobeat wrote: post #193

boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

I work at Google on an open source project and comment on it frequently.

One of the things I really like about working at Google is that they place a lot of trust in the judgement of the individual employees. I generally make it clear when I'm stating my personal opinion versus the "official" (for whatever that means given how informal the project is) one, but I don't have to carefully go through an approved list of talking points, run my HN by the legal department, etc.

Obviously, in certain situations, things get more official and formal. For example, when I went to Google IO to give a talk, we did have some documentation and coaching beforehand about how to handle various questions we might get about non-public stuff, other projects related to ours, etc. We are also expected to run any slides by legal before being publicly shown in a venue with a wide audience like IO. But, even then, the legal folks I've worked with have been a pleasure to talk to.

The company's culture is basically "We hired you because you're smart. We trust you to use your brain." It would be squandering resources to not let their employees use their own intelligence and judgement.

Re: Google Cloud networking issues in us-east1

#228

Post by niyazpk » Tue, Jul 02, 2019, 11:55 PM UTC

mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

As more businesses move their compute to the cloud, one might predict that more people will be impacted by outages in the large cloud providers. This in turn means that the affected people will start up-voting these threads. Expect these to be more common.

Re: Google Cloud networking issues in us-east1

#229

Post by jsjohnst » Tue, Jul 02, 2019, 11:56 PM UTC

lwb wrote: post #131

CaptainJustin wrote: post #101

Earlier quoted context omitted.

It's quite common in cloud solution design to design for failure. One of the common assumptions that we hold to is that one region may go down. Other examples: Assume an instance of an app can go down. Assume a VM can go down. Assume a DC can go down. This is not to excuse the downtime in any way.

Do people ever worry that an entire cloud provider may go down, or is that too unlikely of a case?

In the past ten years:

It’s happened more than once with Azure and GCP. I think it happened once with AWS, but not positive there.

Re: Google Cloud networking issues in us-east1

#230

Post by jsjohnst » Tue, Jul 02, 2019, 11:57 PM UTC

debaserab2 wrote: post #142

lwb wrote: post #131

Earlier quoted context omitted.

Do people ever worry that an entire cloud provider may go down, or is that too unlikely of a case?

I don’t think it’s happened (yet) although some of the earlier outages when AWS was younger were pretty far reaching. I think all of S3 has gone down a time or two.

All of S3 has, but that’s because S3 had a single choke point in a single region for a long time.