Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

171–180 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#171

Post by jkaplowitz »

origami777 wrote: post #122
hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

Most feature rich cloud? I think that title belongs to AWS.

You're right in terms of breadth officially covered. But if you look at the features where they both officially have support, there are many examples where the GCP version is more reliable and usable than the AWS version. Even GKE is an example of this, despite the outage in node pool creation that we're discussing here. Way better than EKS.

(Disclosure: I worked for Google, including GCP, for a few years ending in 2015. I don't work or speak for them now and have no inside info on this outage.)

Re: Google Kubernetes Engine's third consecutive day of service disruption

#172

Post by _asummers »

shiftnight wrote: post #74

I have a question. At what point does k8s make sense? I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well. Monolith for the win! Opinions?

K8s is nice even without microservices. Yeah you don't get nearly the benefits you would in a microservice architecture, but I consider it a control plane for the infrastructure, with an active ecosystem and focus on ergonomics. If you have a really simple infrastructure, you will still need to script spinning up the VMs, setting up the load balancing, etc. but K8s gives you a homogenous layer upon which to put your containers. It's not too much of an overkill, especially with a hosted K8s from e.g. Google, AWS, and soon Digital Ocean and Scaleway.

Things like throwing another node into the cluster, or rolling updates are free, which you would otherwise need to develop yourself. All of that is totally doable, of course, but I like being able to lean on tooling that is not custom, when possible.

When your infrastructure does need to become more complicated, you're already ready for it. Even if I were only serving a single language, starting with a K8s stack makes a lot of sense, to me, from a tooling perspective. Yeah normal VMs might be simpler, conceptually, but I don't consider K8s terribly complicated from a user perspective, when you're staying around the lanes they intend you to stay in. Part of this may also be my having worked with pretty poor ops teams in the past, but I think K8s gives you a really good baseline that gives pretty good defaults about a lot of your infrastructure, without a lot of investment on your part.

That said, if you're managing it on a bare metal server, then VMs may be much easier for you. K8s The Hard Way and similar guides go into how that would work, but managing high availability etcd servers and the like is a bit outside my comfort zone. YMMV.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#173

Post by threeseed »

013a wrote: post #168

Cloud providers have all of the potential in the world to make each region truly isolated. I shouldn't have to architect my application to be multi-cloud, at least for stability reasons. Yet, somehow every major cloud provider experiences global outages. That old AWS S3 outage in us-east-1 was an interesting one; when it went down, many services which rely on S3 also went down, in other regions beside us-east-1 becau…

AWS regions are very much isolated from each other.

We know because we are still waiting here in ap-southeast-2 for services such as EKS to be made available. Pretty sure that any reliance within their backend services on us-east-1 was just a temporary bug and nothing systemic.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#175

Post by ransom1538 »

shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

“2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.”

Ok. So on aws we were* paying for putting systems across regions, but, honestly I don’t get the point. When an entire region is down what I have noticed is that all things are fucked globally on aws. Feel free to pay double - but it seems* if you are paying that much just pay for an additional cloud provider. Looks like it’s the same deal on GCP.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#176

Post by toomuchtodo »

tw04 wrote: post #161
toomuchtodo wrote: post #56

Earlier quoted context omitted.

> I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.

Because they sell themselves as being far more reliable than internal IT. If they weren't selling on uptime, people probably wouldn't be quite so critical of downtime.

As a technology practitioner, it is your failing if you believe them.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#177

Post by human_error »

ransom1538 wrote: post #175
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

“2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.” Ok. So on aws we were* paying for putting systems across regions, but, honestly I don’t get the point. When an entire region is down what I have noticed is that all things are…

> When an entire region is down what I have noticed is that all things are fucked globally on aws.

Do you have an example on this?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#178

Post by ignoramous »

toomuchtodo wrote: post #79
Twirrim wrote: post #66

Earlier quoted context omitted.

(disclaimer: I work for another cloud provider) I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things: 1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumab…

That shouldn't, but they do. S3 goes down [1]. The AWS global console goes down, right after Prime Day outages [2]. Lots of Google Cloud services go down [3, current thread]. Tens of Azure services go down hard [4]. Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and con…

Can't speak for Google, but Facebook and Salesforce chose Cells for HA.

http://highscalability.com/blog/2012/5/9/cell-architectures....

Re: Google Kubernetes Engine's third consecutive day of service disruption

#179

Post by carbocation »

aviv wrote: post #113
carbocation wrote: post #99

Earlier quoted context omitted.

> I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA? I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).

Have not seen any restarts this weekend, and we have several hundred instances on GCE.

Thanks! I'm running Skylake 96 core instances but I haven't given up to try the 64 core instances for comparison yet. If I get another restart, I'll do a 96 vs 64 to try to narrow down the cause. Most likely, of course, this is a software issue on my end, not Google's.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#180

Post by ransom1538 »

human_error wrote: post #177
ransom1538 wrote: post #175

Earlier quoted context omitted.

“2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.” Ok. So on aws we were* paying for putting systems across regions, but, honestly I don’t get the point. When an entire region is down what I have noticed is that all things are…

> When an entire region is down what I have noticed is that all things are fucked globally on aws. Do you have an example on this?

Just grabbed first article. Example: In this case capitalone went down. I don’t work at capitalone - but I imagine they had their data copied across every region 30 times.

https://www.geekwire.com/2018/widespread-outage-amazon-web-s...

Post reply on HN