Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

111–120 of 341 posts

Re: Google Cloud networking issues in us-east1

#111

Post by fkdo »

BurritoAlPastor wrote: post #100
mathattack wrote: post #91

Earlier quoted context omitted.

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

Well, sure, if you hate your devops team and you want to make sure they can’t use any of the proprietary functionality of either provider. At which point, if you want to be managing a fleet of vanilla Linux boxes yourself, why use a cloud provider at all?

Why would you want to lock into a cloud provider? You're losing a lot of operational flexibility for less devops and sysaadmin work.

You are really limiting your tech stack by using standardized things like Jenkins, Docker, K8, mqtt, kafka.

Re: Google Cloud networking issues in us-east1

#112

Post by stingraycharles »

opportune wrote: post #94
gizmo385 wrote: post #90

Earlier quoted context omitted.

> There are a lot of moving pieces in our system and sometimes there are things outside of Google's control. Are you implying that the cause of this outage is not Google's fault? If so, can you go into more details about that?

Not him but oftentimes cloud outages can be due to issues with the network connections to the datacenter, or power outages. Datacenters also sometimes have other single points of failure such as DNS, but those are within the company's control. https://www.networkworld.com/article/3373646/network-problem... https://www.datacenterknowledge.com/uptime/equinix-power-out...

But data centers are typically designed with network and power failures in mind, not? Isn’t this why these kind of ring based network topologies exist, so that whenever a single network connection fails, it can still easily be routed around?

Re: Google Cloud networking issues in us-east1

#113

Post by Decabytes »

tscanausa wrote: post #87
thsowers wrote: post #3

Why so many problems at Google lately? Calendar down two weeks ago[0], and Google Cloud had a larger outage a month ago[1] [0]: https://news.ycombinator.com/item?id=20213092 [1]: https://news.ycombinator.com/item?id=20077421

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside…

Thanks for the reply Terrance. But isn't it more expensive to run in more than one region?

Re: Google Cloud networking issues in us-east1

#114

Post by naniwaduni »

mirceal wrote: post #76
username444 wrote: post #58

Cloudflare was returning a 502 this morning, wonder if they're related. Lots and lots of sites down for about an hour, including all of Shopify.

nope. cloudflare had a bad push / deployment.

"bad push / deployment" seems like it covers 108% of breakage.

Re: Google Cloud networking issues in us-east1

#115

Post by svachalek »

mbrumlow wrote: post #32
zbowling wrote: post #5

Earlier quoted context omitted.

unrelated. very big company with thousands of products that don't suffer outages. two incidents doesn't make a pattern.

So you are making a case for smaller companies run by different people in different ways? So that we don't have huge outages with common systems shared across entire platforms misbehave? When you really care about high availability and security you really don't want all your systems run with the same software, hardware, and coded by the same teams. What does google (or amazon/msft) do to ensure a software echo chambe…

Afaik, these are all as homogenous as they can make them, but there are limits to that. It's hard to move big, old things forward which creates some diversity, but that's probably worse than consistently running the latest stuff everywhere.

If you want heterogeneous environments you have to cobble it together yourself by using multiple services.

Re: Google Cloud networking issues in us-east1

#116

Post by toomuchtodo »

BurritoAlPastor wrote: post #100
mathattack wrote: post #91

Earlier quoted context omitted.

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

Well, sure, if you hate your devops team and you want to make sure they can’t use any of the proprietary functionality of either provider. At which point, if you want to be managing a fleet of vanilla Linux boxes yourself, why use a cloud provider at all?

* You should not be locking yourself into proprietary functionality of a cloud provider unless you are deeply interested in what happened to Oracle customers getting raked over the coals happening to you.

* DevOps teams can be multi-cloud relatively easy when using infrastructure as code tooling (Terraform, Packer, etc) and traditional DevOps practices

* Why manage a fleet of vanilla boxes when you can use vanilla boxes with Kubernetes and not get gouged by cloud providers in the first place?

You don't need to jump off the hype train if you never got on in the first place.

Re: Google Cloud networking issues in us-east1

#117

Post by neonate »

klodolph wrote: post #92
neonate wrote: post #51

Earlier quoted context omitted.

Can you explain that more?

There is no service with 100% availability. You put multiple AZs in one region but nobody was ever pretending that regional failures were impossible, just that single-AZ failures are more common than regional failures. You want high availability, you want multi-regional. Above that you want multi-provider. The same decisions that make regions fail also makes infra-region traffic cheaper. This is true for all large cl…

Thanks, I understand what you meant now.

Re: Google Cloud networking issues in us-east1

#118

Post by cbhl »

dx87 wrote: post #77

Kind of related to this, but these types of outages are why I moved from Google Play to Spotify for streaming music. Their infrastructure seems so large that things that should be a standalone service, like streaming music, are bound to be collateral damage when they mess something up on another service. Having everything provided by one company is convenient until it all goes down at the same time and you can't acce…

In my personal opinion, you should move off of Google Play Music, but not because of the dependency on Google infrastructure.

https://9to5google.com/2018/05/23/google-play-youtube-music-...

https://www.digitaltrends.com/music/what-happens-to-google-p...

Re: Google Cloud networking issues in us-east1

#119

Post by Johnny555 »

user5994461 wrote: post #97
Johnny555 wrote: post #44

Earlier quoted context omitted.

AWS EC2 promises 4 9's (4.3 minutes of downtime/month) before their SLA kicks in, but they only give a 10% discount until availability dips below 99% (7.5 hours of downtime/month) when they give a 30% discount. If availability is below 95% (36 hours) in a month, they give a full refund. For an individual instance, they only promise 90% availability.

Availability of what? I've noticed entire afternoon where it wasn't possible to provision instances of some types, when I was working with AWS daily.

Availability of running instances, I don't think they make any guarantees for availability unreserved on-demand instances (I don't see how they could).

Re: Google Cloud networking issues in us-east1

#120

Post by timc3 »

_wmd wrote: post #104
benbro wrote: post #98

Earlier quoted context omitted.

Is it practical to use several providers when egress is so expensive?

You can move 1.6TB between providers in a month for the same price as a single beefy DB server (m4.16xlarge here). That's a whole lot of logical replication..

You are comparing one overpriced SKU to another over priced SKU.
Post reply on HN