Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

221–230 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#221

Post by haldora »

lilbobbytables wrote: post #189
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

You're doing me a scare. I'm in the evaluation phase with them. Maybe I'm missing something here, but this is not at all what the linked post says. "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI." So, it's a UI console issue, it appears you can still manage "Affected customers can use gcloud command [1] in order to create new Node Pools. [1]" Similarly, it act…

I've been failing all weekend to create nodes in a GKE cluster through either the UI console or gcloud. Even right now I can't get any nodes to spin up.

Edit: I finally got my cluster up and running by removing all nodes, letting it process for a few minutes, then adding new nodes.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#222

Post by davemp »

jeffcore wrote: post #197
davemp wrote: post #138

Earlier quoted context omitted.

I think it’s telling of Google’s culture that the corporate arm felt the need to formalize this in law. I won’t pretend to know what it’s telling. Just suggest that you listen for yourself. Look at rule of law versus the ideas of liberty if you’d like a stronger nudge.

AWS' Customer Agreement[1] essentially has the same language. I wouldn't be surprised to see similar language from other cloud providers as well. Seems rather prudent on their part. [1] https://aws.amazon.com/agreement/

I would suggest the inference I was alluding to would also apply amazon.

Note how I never stated the inference. This is because I wanted to share a way of thinking without feeling the responsibility to reply to people attempting to force me to prove some prescriptive, arbitrary inference rule by exhaustion. I do not participate in such practices casually. I also consider it rude to subject people to such practices without consent. I also believe it is a practice that kills online discussion platforms. See this community’s thought provoking guidelines :)

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#223

Post by navinsylvester »

usmannk wrote: post #118

We had an issue a few weeks ago where the google front-end servers were mangling responses from Pub/Sub and returning 502 responses, making the service completely unusable and knocking over a number of things we have running in production. Despite paying for enterprise support and having in a P1 ticket, we had to spend Friday to Sunday gathering evidence to prove to the support staff that there was indeed a problem,…

In general GCP has quota limits so its expected of customers to catch 5xx and do exponential back off. But this info is not explicitly stated.

From my personal experience - i think all big cloud providers first two level support staff is no good if it isn't an obvious dumb one on your part. I always prefer to forgo support and try to go through every bit of their documentation to figure it out on our own. This helps to save huge amount of time. But if you have developer support - it can help to expedite things little faster though.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#224

Post by scarface74 »

polemic wrote: post #95
shiftnight wrote: post #74

I have a question. At what point does k8s make sense? I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well. Monolith for the win! Opinions?

There's a huge range between monolith and microservice approach, and even a monolith will have dependent services. A simple web stack these days might include nginx, a database, a caching layer, some sort of task broker and then the 'monolith' web app itself. All of that can be sanely managed in k8s.

I definitely wouldn’t be managing my own database or caching layer without a very good reason. I would use a managed service if I were using a cloud provider.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#225

Post by chumboslice »

hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

2 outages in 5 years.

5. Years.

Nothing to see here, move along.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#226

Post by zachberger »

rlancer wrote: post #62
justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

Creating clusters via the UI is still not working for me.

Have you tried via the gcloud command?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#227

Post by gmueckl »

mbrumlow wrote: post #203
mav3rick wrote: post #191

Earlier quoted context omitted.

The work ethic is in tact. It is not fair to load people with stress and ask them to drop everything. You're conflating poor resource allocation with "work ethic". Burning the midnight oil when it can be avoided is not work ethic. The correct way is to load balance outage resolution.

I really don't know how to reply to you as you have set up a bunch of windmills you assumed from my previous post. Who said anything about poor resource allocation? Who said we need to load people with stress? That being said -- when you are on call -- dropping everything is exactly what is expected.

But limiting responsibilities for on call employees is a way of limiting the workload they have during that time - ultimately benefiting the employees. They are on call, not working. I don't see the windmills here.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#228

Post by mav3rick »

gmueckl wrote: post #227
mbrumlow wrote: post #203

Earlier quoted context omitted.

I really don't know how to reply to you as you have set up a bunch of windmills you assumed from my previous post. Who said anything about poor resource allocation? Who said we need to load people with stress? That being said -- when you are on call -- dropping everything is exactly what is expected.

But limiting responsibilities for on call employees is a way of limiting the workload they have during that time - ultimately benefiting the employees. They are on call, not working. I don't see the windmills here.

Yes this was my point.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#229

Post by mav3rick »

mbrumlow wrote: post #203
mav3rick wrote: post #191

Earlier quoted context omitted.

The work ethic is in tact. It is not fair to load people with stress and ask them to drop everything. You're conflating poor resource allocation with "work ethic". Burning the midnight oil when it can be avoided is not work ethic. The correct way is to load balance outage resolution.

I really don't know how to reply to you as you have set up a bunch of windmills you assumed from my previous post. Who said anything about poor resource allocation? Who said we need to load people with stress? That being said -- when you are on call -- dropping everything is exactly what is expected.

Many many AWS people have left citing on call as the worst part of their job. Also, it is really a far fetched allegation that interviewers try to fail interviewees to hide their own incompetence. I get that you may not think much of Google SREs but to allege that is just in bad spirit. I hope you do get to see that the people inside Google are one of the best perks of working here. They are smart and motivated and willing to help each other.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#230

Post by navinsylvester »

ToFab123 wrote: post #216
navinsylvester wrote: post #193

Earlier quoted context omitted.

We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform. So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had ra…

I don't understand why someone would choose to deploy anything mission critical without having an support contract with the ISP, the manufacturer of the the software etc.

I transitioned from collocation to self managed remote server farm and then onto self managed remote vms. All these providers provided de-facto support whether we opted for one or not. You can go to their portal and raise a ticket.

I am not saying with vast numbers its feasible but big cloud providers don't even give you the opportunity to raise a ticket if its their fault. There is a price you pay extra when you opt for any one of them but many don't realize. Having said that - almost all the time, our skilled expertise is better than their initial two level of support staff. We realized it early so we handle it better by going over the documentation and making our code resilient since all cloud platforms have some limit or another since overselling in a region is something they can't avoid. Going multiple regions across when you handle these exceptions is the only way through.

Post reply on HN