Live data from Hacker News

Common mistakes using Kubernetes

blog.pipetail.io

101–110 of 149 posts

Re: Common mistakes using Kubernetes

#101
The guaranteed QoS example in the article is wrong. Kubernetes only sets the Guaranteed QoS if the CPU count is an integer (which 0.5 is not).

Also, to take full benefit of the QoS you need to configure the Kubelet with "--cpu-manager-policy static"[0].

[0]: https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-...

Re: Common mistakes using Kubernetes

#102
post #76

Earlier quoted context omitted.

You can’t have it both ways. If you need to monitor it and take corrective action (which you do) then you shouldn’t rely on it. This is an argument for making your liveness probe == readiness probe. It should just check pod availability in a minimal way, and if continuing to send the pod traffic based on this indicator turns out bad because of congestion, you want to see that causing errors and react, not let the sch…

I don't want the pager to go off just because of some slight non-liveness. That's a likely outcome of high utilization (usually viewed as a good thing, isomorphic with low cost). If you're running really hot and a few tasks are shedding load by playing dead intermittently, that's OK up to a point; if a large portion of pods are doing that at a high rate, that might be bad. You might not even alert on it, just throw i…

> “ I don't want the pager to go off just because of some slight non-liveness.“

That’s just bad engineering. Really, one should want the pager to go off for that and be really pedantic to actually sniff out the root cause and actually fix it.

Hiding that type of issue by letting something like liveness/readiness policy tacitly conceal it is just going to result in a far worse or more systemic issue later with far worse pager disruptions to your life.

You’re skipping flossing every now and then only to need serious root canals later.

Re: Common mistakes using Kubernetes

#103
post #53

Earlier quoted context omitted.

It's quite possible that you have a machine with 192 CPU cores in it, but it's very unlikely that you are able to write a service that scales to that level ... and if you write it in Go it's really unlikely that you can scale even to 8 CPUs. There's nothing weird about having multiple replicas of the same job on the same node. If you look through the Borg traces that Google recently published you can find lots of job…

This is not how defaults work. When you are talking about the realm of the possible , you provide settings that allow you to reach the scenarios that you feel are reasonable, desirable, or lucrative (or commonly enough, some happy combination of the three). Defaults are the realm of the probable. And nobody is requisitioning a 192 core machine without a good bit of due diligence, which would include deciding how to s…

You're suggesting that preventing multiple replicas of the same job to schedule on the same machine as a good default. There's no evidence to support your conclusion, and my experience it quite the opposite. It is much better if people running batch jobs just schedule 100000 tiny replicas, and let the scheduler sort it out. This provides the cluster scheduler with plenty of liquidity. Multiple small processes are more efficient than a shared-nothing single process.

Re: Common mistakes using Kubernetes

#104
post #66
post #44

Earlier quoted context omitted.

Are there good resources for making that decision according to good criteria?

A simple rule of thumb is the number of services you have. For instance, my current employer has been working on getting everything on Kubernetes for the last year or so and we have two services... the frontend server-side renderer, and API.

/snark So is that a cluster per app or a namespace per app?

Re: Common mistakes using Kubernetes

#105
post #70

In my opinion, the most common mistake is not in the article : using kubernetes when you don't need to. Kubernetes has a lot of pros or the papers but in practice it's not worth it for most small and medium companies.

This was my thought exactly. The article is great assuming you need to use k8s, but does leave out the important question: does your project or product require k8s and all the overhead it unavoidably entails? Amazon's Elastic Container Service (ECS) on Fargate deployment type is probably a better option much of the time. Until you maintain your own k8s cluster (including the hosted variants on AWS, GCP, etc.) you mig…

Unless something has changed recently, Fargate is a terrible choice for something that you want up 24x7. IIRC it’s like 2x the cost of similar on demand EC2 instances. It’s better for discrete tasks that can’t run in Lambda because timeout or possibly peak load (although EC2 is probably just as easy if not easier to scale).

But I generally agree with the premise that ECS is a better starting point for people just dipping their toe into containers.

Re: Common mistakes using Kubernetes

#106

I actually disagree with the first recommendation as written - specifically, not to set a CPU resource request to a small amount. It's not always as harmful as it might sound to the novice. It's important to understand that CPU resource requests are used for scheduling and not for limiting. As the author suggests, this can be an issue when there is CPU contention, but on the other hand, it might not be. That's becaus…

1000% agree. Former employer had a proprietary app scheduler that worked like this. We would frequently tell users to request as little CPU as possible. Extra CPU would be shared, but if you made an unreasonable request you’d never get scheduled in the shared environment.

Re: Common mistakes using Kubernetes

#109
post #44

In my opinion, the most common mistake is not in the article : using kubernetes when you don't need to. Kubernetes has a lot of pros or the papers but in practice it's not worth it for most small and medium companies.

Are there good resources for making that decision according to good criteria?

The criteria is basically: do you have the time, people, expertise and willingness to build a custom dev platform on top of Kubernetes?

If yes, Kubernetes might make sense for you, if your company is large enough that a PaaS like Heroku is prohibitively expensive.

If no, you don't want Kubernetes.

https://twitter.com/kelseyhightower/status/93525292372179353...

Re: Common mistakes using Kubernetes

#110
post #70

In my opinion, the most common mistake is not in the article : using kubernetes when you don't need to. Kubernetes has a lot of pros or the papers but in practice it's not worth it for most small and medium companies.

This was my thought exactly. The article is great assuming you need to use k8s, but does leave out the important question: does your project or product require k8s and all the overhead it unavoidably entails? Amazon's Elastic Container Service (ECS) on Fargate deployment type is probably a better option much of the time. Until you maintain your own k8s cluster (including the hosted variants on AWS, GCP, etc.) you mig…

You are right, but talking about whether to use k8s or not would make the article 2-3x longer. It is a totally different topic.

ECS (with Fargate or EC2) is awesome but one might argue that it is still an overkill for a lot of people.

Some people look for scaling, immutability, effective resource utilization and they think they NEED containers. That is not entirely true. A lot of people would do totally ok with one EC2 instance.

Either way, kubernetes is awesome, there is a big community behind it maintaining a lot of tooling, docs, how-to guides, etc. That is very valuable for a lot of people too :)

Post reply on HN