Google Cloud networking issues in us-east1 • Hacker News Board

Re: Google Cloud networking issues in us-east1

#171

Post by Spooky23 » Tue, Jul 02, 2019, 9:19 PM UTC

lwb wrote: post #131

CaptainJustin wrote: post #101

Earlier quoted context omitted.

It's quite common in cloud solution design to design for failure. One of the common assumptions that we hold to is that one region may go down. Other examples: Assume an instance of an app can go down. Assume a VM can go down. Assume a DC can go down. This is not to excuse the downtime in any way.

Do people ever worry that an entire cloud provider may go down, or is that too unlikely of a case?

It’s a legit concern, but it adds complexity that will probably cause more outages than the thing you are worried about.

IMO, you’re better off with a private data center or colo and separate integrations with cloud.

Re: Google Cloud networking issues in us-east1

#172

Post by Johnny555 » Tue, Jul 02, 2019, 9:22 PM UTC

itslennysfault wrote: post #169

Johnny555 wrote: post #44

Earlier quoted context omitted.

AWS EC2 promises 4 9's (4.3 minutes of downtime/month) before their SLA kicks in, but they only give a 10% discount until availability dips below 99% (7.5 hours of downtime/month) when they give a 30% discount. If availability is below 95% (36 hours) in a month, they give a full refund. For an individual instance, they only promise 90% availability.

What a craptactular SLA.

Is there any cloud provider with a better SLA? Google's is similar, but only gives a 50% credit if the service has less than 95% availability.

Re: Google Cloud networking issues in us-east1

#173

Post by autoexec » Tue, Jul 02, 2019, 9:23 PM UTC

username444 wrote: post #58

Cloudflare was returning a 502 this morning, wonder if they're related. Lots and lots of sites down for about an hour, including all of Shopify.

It's good we've built this massive decentralized network to withstand even major nuclear attacks only to have massive parts of it fail because we've put so much in a few centralized and failable hands.

Re: Google Cloud networking issues in us-east1

#174

Post by tjr225 » Tue, Jul 02, 2019, 9:25 PM UTC

tomcam wrote: post #127

toomuchtodo wrote: post #116

Earlier quoted context omitted.

* You should not be locking yourself into proprietary functionality of a cloud provider unless you are deeply interested in what happened to Oracle customers getting raked over the coals happening to you. * DevOps teams can be multi-cloud relatively easy when using infrastructure as code tooling (Terraform, Packer, etc) and traditional DevOps practices * Why manage a fleet of vanilla boxes when you can use vanilla bo…

if I voluntarily choose a provider at a price that’s acceptable to me am I being gouged?

Not yet, but it seems obvious to me that the GP was referring to a situation where the price changes and then you are getting gouged. That's exactly what the negative connotations of lock-in refer to.

Re: Google Cloud networking issues in us-east1

#175

Post by tyingq » Tue, Jul 02, 2019, 9:25 PM UTC

vinay_ys wrote: post #81

foobiekr wrote: post #41

Earlier quoted context omitted.

It’s surprisingly hard to avoid shared fate links and it’s one of the things I would have thought google would be expert at.

It's not that hard. In India because of so much construction related digging cuts OFCs, we do the path planning quite well and our redundancies get tested quite regularly whether you want to or not.

It can be hard. Getting redundant separated paths under/over railroad tracks, for example, might require political power that not everyone has. Google, of course, has plenty.

Re: Google Cloud networking issues in us-east1

#176

Post by StreamBright » Tue, Jul 02, 2019, 9:27 PM UTC

lwb wrote: post #131

CaptainJustin wrote: post #101

Earlier quoted context omitted.

It's quite common in cloud solution design to design for failure. One of the common assumptions that we hold to is that one region may go down. Other examples: Assume an instance of an app can go down. Assume a VM can go down. Assume a DC can go down. This is not to excuse the downtime in any way.

Do people ever worry that an entire cloud provider may go down, or is that too unlikely of a case?

The chance that AWS goes down is much smaller than anything else going down. There are many SPOFs in a typical smaller company setup, most of those are not even obvious to the operators.

Re: Google Cloud networking issues in us-east1

#177

Post by [deleted] » Tue, Jul 02, 2019, 9:27 PM UTC

boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

[deleted]

Re: Google Cloud networking issues in us-east1

#178

Post by Spooky23 » Tue, Jul 02, 2019, 9:27 PM UTC

tomcam wrote: post #127

toomuchtodo wrote: post #116

Earlier quoted context omitted.

* You should not be locking yourself into proprietary functionality of a cloud provider unless you are deeply interested in what happened to Oracle customers getting raked over the coals happening to you. * DevOps teams can be multi-cloud relatively easy when using infrastructure as code tooling (Terraform, Packer, etc) and traditional DevOps practices * Why manage a fleet of vanilla boxes when you can use vanilla bo…

if I voluntarily choose a provider at a price that’s acceptable to me am I being gouged?

Each provider will seek to make you take their one true path, or you need to do your own engineering.

Using the providers path isn’t necessarily gouging, but it isn’t cost optimized either. The answer depends on you.

That said, cloud is like any tenant/landlord relationship. Your rights are linked to time and are whatever your contract provides. If you didn’t like Office 2007, you didn’t buy it. If you don’t like Office 365, 2021 edition, too bad.

Re: Google Cloud networking issues in us-east1

#179

Post by [deleted] » Tue, Jul 02, 2019, 9:28 PM UTC

mathattack wrote: post #91

tscanausa wrote: post #87

Earlier quoted context omitted.

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside…

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

[deleted]

Re: Google Cloud networking issues in us-east1

#180

Post by peterwwillis » Tue, Jul 02, 2019, 9:29 PM UTC

BurritoAlPastor wrote: post #100

mathattack wrote: post #91

Earlier quoted context omitted.

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

Well, sure, if you hate your devops team and you want to make sure they can’t use any of the proprietary functionality of either provider. At which point, if you want to be managing a fleet of vanilla Linux boxes yourself, why use a cloud provider at all?

> why use a cloud provider at all?

Cost+speed of scalability, and managed services. If you rarely need to scale, your workloads are all predictable, and you don't need managed services/support, you should just buy some VPSes or dedicated boxes.