Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

61–70 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#61

Post by trhway »

jasonlotito wrote: post #25
marcinzm wrote: post #22

>Nov 09, 2018 05:59 >We will provide more information by Monday, 2018-11-12 11:00 US/Pacific. Wait, did the people tasked with fixing this just take the weekend off?

The people tasked with fixing this aren't the ones providing the updates.

They have whole book describing who fixes, provides updates, etc. Fun meditative reading while waiting for the outage to get fixed.

https://landing.google.com/sre/sre-book/chapters/managing-in...

Looks like this time Mary took the whole week off without telling Josephine :)

Re: Google Kubernetes Engine's third consecutive day of service disruption

#62

Post by rlancer »

justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

Creating clusters via the UI is still not working for me.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#64

Post by raincom »

closeparen wrote: post #8

Doesn’t GKE “just” run an independent Kubernetes cluster on customer VMs? How is a widespread outage like this possible?

Nope, GKE = master/control plane owned by Google. Customers are just tenants, who can schedule workloads.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#65

Post by noselasd »

tehlike wrote: post #59
hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

2 outage in 5 years sounds pretty low, to be honest. Disclaimer: google employee in ads, who worked on many many fires throughout the years, but talking from my personal perspective and not from my employer. I am sure we are striving to have 0, but realistically, i have seen many that says things happen. Learn, and improve.

The issue people have with it is that it's global, not regional, indicating that there are dependencies in the entire architecture that people does not expect to be there.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#66

Post by Twirrim »

toomuchtodo wrote: post #56
hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

> I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.

(disclaimer: I work for another cloud provider)

I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things:

1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it.

or:

2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#67

Post by cloakandswagger »

geggam wrote: post #30
scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Infrastructure as code. Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services. This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

Totally impractical for any small business and of questionable usefulness for large ones. You'd be giving up the largest benefit of platforms like AWS--ready to use services for common tasks--to avoid the infinitesimally small chance of AWS having some doomsday global outage.

To answer the original question: It looks like this issue was just a UI bug that affected the console, the service itself wasn't impacted. Events that do impact the service will be contained to a region, meaning you can mitigate it with proper redundancy across regions, no zany multi-cloud solution required.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#68

Post by kenan_warren »

regnerba wrote: post #10
closeparen wrote: post #8

Doesn’t GKE “just” run an independent Kubernetes cluster on customer VMs? How is a widespread outage like this possible?

GKE does the creation of the VMs and setup of them, joining them to the cluster and applying labels for example. The specific issue appears to be about creating new "node pools". Creating standard VMs in GCP works fine however, so this is specific to GKE and their internal tooling that integrates with the rest of GCP. GKE doesn't (at least to my knowledge) allow you to create VMs separately and join them to the clust…

It's actually not just GKE, there have been issues creating normal VMs since late Friday night. It seems anything that required creating VMs gave back resource exhaustion errors. I finally got a cluster for us-east1 setup last night so it looks like the resource issues are clearing up though.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#69

Post by manigandham »

toomuchtodo wrote: post #56
hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

> I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.

The major issue is that outages are global instead of regional, effectively making it impossible to design around using the typical region/zone redundancy.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#70

Post by INTPenis »

jasonlotito wrote: post #25
marcinzm wrote: post #22

>Nov 09, 2018 05:59 >We will provide more information by Monday, 2018-11-12 11:00 US/Pacific. Wait, did the people tasked with fixing this just take the weekend off?

The people tasked with fixing this aren't the ones providing the updates.

Understandable but in my experience the incident manager assigned is still supposed to keep track of progress during weekends when you have a major incident.

And this is likely a major incident with significant customer impact.

The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.

Post reply on HN