Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

41–50 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#41

Post by ben_jones »

marcinzm wrote: post #36
justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

I appreciate all the effort you're putting in and I understand such situations can be stressful but user's having to depend on someone responding on hacker news for status updates seems really amateur for an organization the size of google.

As much as I love bashing big corps I see HN as a supplementary communication channel for products like GCP - its a luxury we get to access alongside normal customer support channels in the GCP console, twitter, etc.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#42

Post by scarface74 »

dgemm wrote: post #31
scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Assuming you have architected multi-region I'm not sure how realistic that scenario is. AWS regions are mostly standalone, I have seen services go down in a region but never globally.

The most recent example that I can think of when something went down globally was Route 53 - the one service that AWS promises 100% up time for.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#43

Post by thwy12321 »

ben_jones wrote: post #41
marcinzm wrote: post #36

Earlier quoted context omitted.

I appreciate all the effort you're putting in and I understand such situations can be stressful but user's having to depend on someone responding on hacker news for status updates seems really amateur for an organization the size of google.

As much as I love bashing big corps I see HN as a supplementary communication channel for products like GCP - its a luxury we get to access alongside normal customer support channels in the GCP console, twitter, etc.

Critical service is failing, minimal information about why, but we should be so happy someone says a few sentences on here? For all of the engineering elitism coming out of google, Amazon is way more on their game across a number of products.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#44

Post by mey »

scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

First, determine your tolerance. How much does downtime per min cost you? How long can you be down? A lot of the time it may be cheaper to apologize to your customers than build a truly reliable system.

Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage?

There are formal academic ways to determine what your overall availability is, but don't have those on hand. Suffice to say, it takes significant research, planning, execution, and testing to ensure a target availability. (See Netflix https://medium.com/netflix-techblog/the-netflix-simian-army-... ) if someone says they have 99.9% or better up time, they had better have proof in my mind (or a fat SLA violation payout)

People outsource to cloud providers not because they are cheap, but because managing infra in house is hard. Also move fast and break things.

Read AWS docs about availability, there are availability zones in a region, spread across those to minimize impact. Then test when something goes down. Fix/repeat.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#45

Post by hacknat »

eknkc wrote: post #39
scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

This might be an unpopular opinion but, Going multi region on AWS should be safe enough. If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days. You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not fee…

Yeah, AWS has never had a global outage, Google has had 2 now.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#46

Post by justinsb »

marcinzm wrote: post #22

>Nov 09, 2018 05:59 >We will provide more information by Monday, 2018-11-12 11:00 US/Pacific. Wait, did the people tasked with fixing this just take the weekend off?

The incident with the UI (where we suggested using gcloud temporarily) was opened in https://status.cloud.google.com/incident/container-engine/18..., but then what sure looks to me like the same incident was closed in https://status.cloud.google.com/incident/container-engine/18....

My working assumption is that 18006 should have closed out 18005. But now it sounds like there's a different issue, which we're working to get to the bottom of.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#47

Post by viraptor »

scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

> What would a small business do as a contingency plan?

Depends on your definition of small. If it's small enough not to have a dedicated infrastructure team designing multicloud solution, then the contingency plan may be: switch DNS to a static site saying "we're down until AWS fixes the issue, check back later".

Otherwise it depends on your specific scenario, your support contracts, and lots of other things. You need to decide what matters, how much the mitigation costs vs downtime, and go from there.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#48

Post by aviv »

scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

We invested early in being multi-region on GCP as well as multi-cloud with AWS as a fully redundant option if it ever became necessary to fail over to them.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#49

Post by Masiosare »

pfd1986 wrote: post #18
thwy12321 wrote: post #16

Been trying to spin up vm instances all day, had to try every single zone just to get one up. Not only is this incredibly harmful to a technology business dependent on this infra, it wasnt obvious to me what the issue was until I tried creating instances. Nothing says, hey resources are constrained here, try this one. Just about ready to bite the bullet and move to aws.

Same here. We have spent 2 days trying to create instances and migrate images just to figure out later they can't start. Right when I convinced our project to get migrated from AWS...

Same question... why would you do that? AWS is super stable most of the time. I have been running k8s over EC2 (not eks) for a year and works like a charm. I've even run experiments using spot instances and it's pretty good (no guarantee there of course).

Re: Google Kubernetes Engine's third consecutive day of service disruption

#50

Post by justinsb »

Jedi72 wrote: post #21

"The data says engagement is down 46%, I think its time we drop the product." - Someone at Google right now, probably.

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy (https://cloud.google.com/terms/), whose intent is to give you some assurances.

(I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

Post reply on HN