Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

111–120 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#111

Post by aviv »

antpls wrote: post #91
justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the…

We have created GCE instances in several US regions without any issue today. Last one was 10 minutes ago in west2.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#112

Post by whydoineedthis »

justinsb wrote: post #50
Jedi72 wrote: post #21

"The data says engagement is down 46%, I think its time we drop the product." - Someone at Google right now, probably.

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy ( https://cloud.google.com/terms/ ), whose intent is to give you some assurances. (I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

im pretty sure he just forget the /s (sarcasm) on his post, but this was pretty cool information anyway, so thanks!

Re: Google Kubernetes Engine's third consecutive day of service disruption

#113

Post by aviv »

carbocation wrote: post #99
antpls wrote: post #91

Earlier quoted context omitted.

The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the…

> I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA? I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).

Have not seen any restarts this weekend, and we have several hundred instances on GCE.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#114

Post by arachnophobe »

gaius wrote: post #51
mey wrote: post #44

Earlier quoted context omitted.

First, determine your tolerance. How much does downtime per min cost you? How long can you be down? A lot of the time it may be cheaper to apologize to your customers than build a truly reliable system. Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage? There are formal academic w…

If all your competitors are down too, the equation changes, no?

Yep - If all my competitors are down the equation changes such that I _definitely_ want to be up.

Any trade comes to me if it's urgent, and I appear more professional as I've got a functioning system.

I might be an chancer running my entire system off an shoestring but being up when everyone else has taken a dive looks good.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#115

Post by Twirrim »

ramphastidae wrote: post #101
izacus wrote: post #57

Earlier quoted context omitted.

It's weekend, why wouldn't you take it off? It's just silly software.

A lot of people's businesses, reputations and livelihoods depend on "silly software," not to mention that they are paying customers themselves.

I know sarcasm doesn't translate well on the internet, but how the heck did you manage to miss it in OPs post?!

Re: Google Kubernetes Engine's third consecutive day of service disruption

#116

Post by Twirrim »

mcqueenjordan wrote: post #97
jorams wrote: post #86

Earlier quoted context omitted.

I'm not sure where you get this idea, but AWS has definitely had global outages. 1.5 years ago there were massive global issues with S3, causing even their own status dashboard to malfunction. edit: I stand corrected. Apparently the S3 outage wasn't global, though its effects were. Meanwhile, this outage has only really been noticeable to ops teams, since it doesn't affect existing nodes or anything outside GKE. It's…

On February 28th, 2017, S3 had issues in us-east-1, not globally. It just happens that most customers create their buckets in us-east-1. (It’s the default bucket creation region.)

Buckets creation/deletion is a global operation, namespaces are all global for it. When us-east-1 goes down, so does bucket creation. All other operations proceed merrily, but yeah as we see over and over again, whenever us-east-1 has an outage everything seems to shit the bed because they've built everything out in us-east-1. Hopefully us-east-2 is starting to eat in to that.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#117

Post by Twirrim »

xchaotic wrote: post #35
geggam wrote: post #30

Earlier quoted context omitted.

Infrastructure as code. Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services. This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

I think you are downplaying minimal

I don't think OP was intending "minimal" to mean it would be easy to get to the stage where it's possible, just that once you've got all your infrastructure-as-code stuff set up correctly, you ought to be able to just be pressing buttons / running scripts and have your infrastructure up and running in another cloud provider.

Even when working in small companies with small infrastructure, I've kept recreation of infrastructure as one of my high priorities (one reason it really bugged me in one job to have to depend on Oracle Databases that I couldn't automate to the same degree.)

In my mind, it's not different from the importance of having, and testing restoration of, backups. If your infrastructure gets compromised somehow, or you find yourself up the creek with your provider, you've got to be able to rebuild everything from scratch.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#118

Post by usmannk »

We had an issue a few weeks ago where the google front-end servers were mangling responses from Pub/Sub and returning 502 responses, making the service completely unusable and knocking over a number of things we have running in production. Despite paying for enterprise support and having in a P1 ticket, we had to spend Friday to Sunday gathering evidence to prove to the support staff that there was indeed a problem, because their monitoring wasn't detecting it. Right now I'm doing something similar (and since Friday!) but for TLS issues they're having. Again, because their support reps don't believe there's a problem. There are so many more problems than they ever show on their status page...

Re: Google Kubernetes Engine's third consecutive day of service disruption

#119

Post by rorykoehler »

joshuamorton wrote: post #93
INTPenis wrote: post #70

Earlier quoted context omitted.

Understandable but in my experience the incident manager assigned is still supposed to keep track of progress during weekends when you have a major incident. And this is likely a major incident with significant customer impact. The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.

Incident manager isn't public comms person. The person updating that status dashboard may or may not be an engineer, the IM certainly is.

They should take a few mins out of their weekend to update the dashboard regardless. If my small 25 employee company can do it Google can do it.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#120

Post by michaelt »

joshuamorton wrote: post #93
INTPenis wrote: post #70

Earlier quoted context omitted.

Understandable but in my experience the incident manager assigned is still supposed to keep track of progress during weekends when you have a major incident. And this is likely a major incident with significant customer impact. The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.

Incident manager isn't public comms person. The person updating that status dashboard may or may not be an engineer, the IM certainly is.

At a smaller, less corporate company the engineer/public comms person dividing line would not be so ossified the divide couldn't be bridged when the situation called for it.
Post reply on HN