Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Common mistakes using Kubernetes

blog.pipetail.io

Post reply on HN
Print view View original

141–149 of 149 posts

Re: Common mistakes using Kubernetes

#141

Post by yongjik »

user5994461 wrote: post #121
yongjik wrote: post #96

Well, nobody asked me, and I'm no expert, but here's my list of what (not) to do in Kubernetes (if I had the authority). 1. There. Is. No. Machine. (Insert matrix meme here.) Before you open up your cluster to the rest of company, drill it down to them. Maybe even create a Google Form where they have to sign "I hereby acknowledge that there is no machine in k8s and any attempt to tie my job to a particular machine me…

4. Centralized logging is the basics in any company. A container simply logs to stdout (kubernetes) and a fluentd/logstash agent can forward it. 7. Sadly if existing software can't run in kubernetes, this severely limit the benefits of having kubernetes, why use something that can't be used? If the jobs already have service discovery, they might be better off running on hostNetwork or whatever allow them to work as i…

Re: 7, if you use a separate discovery like zookeeper, then it will have its own idea of which jobs are alive and ready, and k8s will have its own idea of the same thing - they may not necessarily agree with each other. They don't even know the existence of each other. Maybe it would work, but it seems like more hassle than rewriting the offending part.

From what I've seen, Kubernetes is a quintessential Google product - it has a very particular idea of how jobs should be run, and the farther you stray away from it, the more it will cost your sanity. If that curtails the benefits of k8s for you, then yes, I think you should revisit whether you really need k8s, based on that limitation.

Just my two cents.

Re: Common mistakes using Kubernetes

#142

Post by rwoll »

marekaf wrote: post #110
rwoll wrote: post #70

Earlier quoted context omitted.

This was my thought exactly. The article is great assuming you need to use k8s, but does leave out the important question: does your project or product require k8s and all the overhead it unavoidably entails? Amazon's Elastic Container Service (ECS) on Fargate deployment type is probably a better option much of the time. Until you maintain your own k8s cluster (including the hosted variants on AWS, GCP, etc.) you mig…

You are right, but talking about whether to use k8s or not would make the article 2-3x longer. It is a totally different topic. ECS (with Fargate or EC2) is awesome but one might argue that it is still an overkill for a lot of people. Some people look for scaling, immutability, effective resource utilization and they think they NEED containers. That is not entirely true. A lot of people would do totally ok with one E…

> You are right, but talking about whether to use k8s or not would make the article 2-3x longer. It is a totally different topic.

Sorry—my comment was written poorly. I think the article is good as is and agree k8s or not is a different topic. My comment intended to be a reminder for all of us reading it that we should ask more questions before choosing a technology.

I’ve found a trend in engineering (including with myself), we sometimes forget to take a step back and see the forest.

Re: Common mistakes using Kubernetes

#143

Post by hinkley »

jeffbee wrote: post #140
hinkley wrote: post #138

Earlier quoted context omitted.

Still the same question. Do you think that batch processing is the default activity in Kubernetes, or something that people find after they are familiar with the system?

Yes, I think batch workloads are the most common workloads, in resource-weighted terms, among k8s users.

You're being slippery, which comes across as dishonest.

Why does the resource weight have anything to do with the choice of defaults? Settings don't care how often they are read, they only care how often they are set. Large jobs use a disproportionate amount of total resources, sure, but they are tiny uptick in total configuration.

The stakes are higher, but so is the 'budget' for getting things right. I can deploy 5 servers and just wait to see what happens. If I'm doing an overnight job to process a billion records, I'd better be doing some due diligence beforehand, or I have nobody to blame but me. And the failure mode here is that I didn't spend money fast enough to get the job done.

With the current defaults what happens is I blow my monthly budget in one night. Which is very convenient for the vendor, but not convenient for my company.

"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair

Re: Common mistakes using Kubernetes

#144

Post by snupples »

yongjik wrote: post #96

Well, nobody asked me, and I'm no expert, but here's my list of what (not) to do in Kubernetes (if I had the authority). 1. There. Is. No. Machine. (Insert matrix meme here.) Before you open up your cluster to the rest of company, drill it down to them. Maybe even create a Google Form where they have to sign "I hereby acknowledge that there is no machine in k8s and any attempt to tie my job to a particular machine me…

These are all excellent ideas and honestly better than the article. You're being too modest.

We solve #2 (and more) by having a highly restrictive default PodSecurityPolicy. We started with a combination of GCE and OpenShift default examples (although we use neither) which are published on github. PSP lets admins and security relax a little at night.

Anything #3 is solved by setting highly restrictive LimitRanges in each namespace that must be overridden in the deployment specs.

It's not mandatory that you override the defaults, but if you don't you're going to get poor performance. If you're fine with that the admins are also fine with that.

Sometimes some occasional throttling during spikes is totally acceptable - it all just depends on whether the app actually needs maximum performance. There are many other apps in the cluster that DO, and in the grand scheme of things they benefit by having the ones that DON'T get throttled.

Apparently Borg handles CPU differently (and better) than k8s in the multitenancy model, but this is the best "poor man's" borg I can come up with and it works for us.

Re: Common mistakes using Kubernetes

#145

Post by user5994461 »

yongjik wrote: post #141
user5994461 wrote: post #121

Earlier quoted context omitted.

4. Centralized logging is the basics in any company. A container simply logs to stdout (kubernetes) and a fluentd/logstash agent can forward it. 7. Sadly if existing software can't run in kubernetes, this severely limit the benefits of having kubernetes, why use something that can't be used? If the jobs already have service discovery, they might be better off running on hostNetwork or whatever allow them to work as i…

Re: 7, if you use a separate discovery like zookeeper, then it will have its own idea of which jobs are alive and ready, and k8s will have its own idea of the same thing - they may not necessarily agree with each other. They don't even know the existence of each other. Maybe it would work, but it seems like more hassle than rewriting the offending part. From what I've seen, Kubernetes is a quintessential Google produ…

If we're talking about zookeeper for service discovery. The registering application can maintain a connection to zookeeper and ping every few seconds. So zookeeper has a very good idea of what's running or not, usually more accurate than kubernetes.

Actually, zookeeper has API to register only when ready and to listen to events/changes. Not sure kubernetes has equivalent stable API so might be hard to port over.

A rewrite might be a solution, except it's not because it's doomed to fail. We're discussing service discovery, which implies multiple clients and servers, probably managed by different teams and written in different languages. The odds of completing a coordinated rewrite effort are abysmal. ^^

Well. I am thinking out loud. It's a real problem my company was facing. We've got kubernetes clusters that are supposed to run applications and we've got apps using zookeeper that can't run in kubernetes because it breaks the service discovery. I will eventually have to hint people how to make it happen, after one year of kubernetes hardly going anywhere.

Re: Common mistakes using Kubernetes

#146

Post by Leonid99 »

fnord123 wrote: post #22
xrd wrote: post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

Most books frame this as "one common mistake is ...". I always understood this phrase to mean that the author has personally made this mistake more than once.

Re: Common mistakes using Kubernetes

#147

Post by kerny »

tomnipotent wrote: post #12
haolez wrote: post #3

... requiredDuringSchedulingIgnoredDuringExecution: ... This instantly remembered me of this: https://thedailywtf.com/articles/the-longest-method Kubernetes sometimes shows its Java roots.

To be fair, I don't see how this could be shorter in any other language without losing readability.

They could have used: soft, hard and strict with the documentation explaining the differences.

Re: Common mistakes using Kubernetes

#148

Post by apple4ever »

LambdaB wrote: post #49
apple4ever wrote: post #47

"more tenants or envs in shared cluster" This is what I'm trying to convince my current company about. They want everything in a single cluster (prod, test, stage, qa). Of course self hosting makes this more difficult to justify, since it is additional expenses for more machines.

Have you considered using OpenShift instead of Kubernetes? It comes with vastly improved multitenancy features, as well as other aspects, in regards to plain Kubernetes. OKD, the open sourced package of OpenShift allows full self-hosting: https://www.okd.io

OpenShift comes with its own headaches from my understanding. And we are too deep into Kube to switch now.

Re: Common mistakes using Kubernetes

#149

Post by apple4ever »

moondev wrote: post #51
apple4ever wrote: post #47

"more tenants or envs in shared cluster" This is what I'm trying to convince my current company about. They want everything in a single cluster (prod, test, stage, qa). Of course self hosting makes this more difficult to justify, since it is additional expenses for more machines.

That sounds like a disaster waiting to happen! Seems like a perfect use-case for Cluster API: https://cluster-api.sigs.k8s.io/user/quick-start.html Have one global "mgmt cluster" with several workload clusters

You bet it is! Haha

Ah interesting! We'll have to look into that.

Post reply on HN