Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

51–60 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#51

Post by gaius »

mey wrote: post #44
scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

First, determine your tolerance. How much does downtime per min cost you? How long can you be down? A lot of the time it may be cheaper to apologize to your customers than build a truly reliable system. Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage? There are formal academic w…

If all your competitors are down too, the equation changes, no?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#52

Post by hacknat »

Question to Google employees:

Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich cloud (particularly your networking products), but down time like this is unacceptable.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#54

Post by thaumasiotes »

eknkc wrote: post #39
scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

This might be an unpopular opinion but, Going multi region on AWS should be safe enough. If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days. You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not fee…

> If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.

Additionally, from a "nobody ever got fired for buying IBM" perspective, you're unlikely to catch much blame from your users for going down when everyone else was down too.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#55

Post by cthalupa »

scarface74 wrote: post #42
dgemm wrote: post #31

Earlier quoted context omitted.

Assuming you have architected multi-region I'm not sure how realistic that scenario is. AWS regions are mostly standalone, I have seen services go down in a region but never globally.

The most recent example that I can think of when something went down globally was Route 53 - the one service that AWS promises 100% up time for.

Are you thinking of the event in April? That was not a Route 53 outage, that was a BGP hijack.

https://blog.thousandeyes.com/amazon-route-53-dns-and-bgp-hi...

Re: Google Kubernetes Engine's third consecutive day of service disruption

#56

Post by toomuchtodo »

hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

> I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective.

It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#57

Post by izacus »

marcinzm wrote: post #26
jasonlotito wrote: post #25

Earlier quoted context omitted.

The people tasked with fixing this aren't the ones providing the updates.

Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.

It's weekend, why wouldn't you take it off? It's just silly software.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#58

Post by justinsb »

franky_g wrote: post #53

Had it affected all regions or just some? Is there another status page Google? Coz the last update I'm looking at...is dated on the 9th..

The general page is at https://status.cloud.google.com/; you can scroll down to see GKE, and my (unofficial) belief is that https://status.cloud.google.com/incident/container-engine/18... should have closed out https://status.cloud.google.com/incident/container-engine/18...

_If_ that's the case, something else is causing the error messages other people are seeing

Re: Google Kubernetes Engine's third consecutive day of service disruption

#59

Post by tehlike »

hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

2 outage in 5 years sounds pretty low, to be honest.

Disclaimer: google employee in ads, who worked on many many fires throughout the years, but talking from my personal perspective and not from my employer. I am sure we are striving to have 0, but realistically, i have seen many that says things happen. Learn, and improve.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#60

Post by _wmd »

hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

It is a natural effect of building massive yet flat homogeneous systems, failures tend to be greatly amplified.

Most of what you can read of Google's approach will teach you their ideal computing environment is a single planetary resource, pushing any natural segmentation and partitioning out of view.

Post reply on HN