Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Common mistakes using Kubernetes

blog.pipetail.io

Post reply on HN
Print view View original

51–60 of 149 posts

Re: Common mistakes using Kubernetes

#51

Post by moondev »

apple4ever wrote: post #47

"more tenants or envs in shared cluster" This is what I'm trying to convince my current company about. They want everything in a single cluster (prod, test, stage, qa). Of course self hosting makes this more difficult to justify, since it is additional expenses for more machines.

That sounds like a disaster waiting to happen!

Seems like a perfect use-case for Cluster API: https://cluster-api.sigs.k8s.io/user/quick-start.html

Have one global "mgmt cluster" with several workload clusters

Re: Common mistakes using Kubernetes

#52

Post by jeffbee »

otterley wrote: post #42

I actually disagree with the first recommendation as written - specifically, not to set a CPU resource request to a small amount. It's not always as harmful as it might sound to the novice. It's important to understand that CPU resource requests are used for scheduling and not for limiting. As the author suggests, this can be an issue when there is CPU contention, but on the other hand, it might not be. That's becaus…

If you ask for 0.001 CPU share, you might get it. I would advise caution. You that pod gets scheduled on a node with another node that asks for 4 CPUs and 100MB of memory, it's not going to get any time.

Re: Common mistakes using Kubernetes

#53

Post by jeffbee »

hinkley wrote: post #36

> You can't expect kubernetes scheduler to enforce anti-affinites for your pods. You have to define them explicitly. Why isn't this the default behavior? Why don't I have to go in and tell it that it's okay to have multiple instances on the same node? Why? So that I somehow feel like I've contributed to the whole process by fixing something that never should break in the first place? I know of a few pieces of code wh…

It's quite possible that you have a machine with 192 CPU cores in it, but it's very unlikely that you are able to write a service that scales to that level ... and if you write it in Go it's really unlikely that you can scale even to 8 CPUs. There's nothing weird about having multiple replicas of the same job on the same node. If you look through the Borg traces that Google recently published you can find lots of jobs with multiple replicas per node.

Re: Common mistakes using Kubernetes

#54

Post by gridlockd »

fnord123 wrote: post #22
xrd wrote: post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

> Your editor was very fucking wrong.

The editor is completely right in what they were saying. You just want them to be wrong, because you'd prefer to live in the fantasy world where they are wrong.

Let's say you go to get a surgery. You don't want the doctor to tell you about all the times they fucked up and what the awful consequences were. It doesn't matter that they're probably a better surgeon now, having learned from their mistakes. Psychologically, you need that person with the sharp tool poking around inside your body to be a superhuman.

To a lesser degree, the same is true for any expert. Of course everybody makes mistakes. Notice the de-personalization in the word "everybody". You can talk about the mistakes everybody makes, or those ones that many people make. If you talk about your own mistakes however, you lose the superhuman status. There may be a few situations where that somehow helps you, but not when you want to sell books.

Re: Common mistakes using Kubernetes

#55

Post by otterley »

jeffbee wrote: post #52
otterley wrote: post #42

I actually disagree with the first recommendation as written - specifically, not to set a CPU resource request to a small amount. It's not always as harmful as it might sound to the novice. It's important to understand that CPU resource requests are used for scheduling and not for limiting. As the author suggests, this can be an issue when there is CPU contention, but on the other hand, it might not be. That's becaus…

If you ask for 0.001 CPU share, you might get it. I would advise caution. You that pod gets scheduled on a node with another node that asks for 4 CPUs and 100MB of memory, it's not going to get any time.

It depends. If the second pod requests 4 CPUs, it doesn't necessarily mean that the first pod can't use all the CPUs in the uncontended case.

A lot of this depends on policy and cooperation, which is true for any multitenant system. If the policy is that nobody requests CPU, then the behavior will be like an ordinary shared Linux server under load - the scheduler will manage it as fairly as possible. OTOH, if there are pods that are greedy and pods that are parsimonious in terms of their requests, the greedy pods will get the lion's share of the resources if it needs them.

The flip side of overallocating CPU requests is cost. This value is subtracted from the available resources, making the node unavailable to do other useful work. Most of the time I see customers making the opposite mistake - overallocating CPU requests so much that their overall CPU utilization is well under 25% during peak periods.

Re: Common mistakes using Kubernetes

#56

Post by jeffbee »

mlthoughts2018 wrote: post #27

I think there is more to the story for some of these points and it can be dangerous to just take this at face value of best practices. For example on the liveness / readiness probe item, the article says, > “ The other one is to tell if during a pod's life the pod becomes too hot handling too much traffic (or an expensive computation) so that we don't send her more work to do and let her cool down, then the readiness…

You of course should monitor the rate of liveness flapping for your services. The need to monitor it does not imply that it's a bad feature.

Re: Common mistakes using Kubernetes

#58

Post by mgkimsal »

xrd wrote: post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

personally... I'd like to see that sort of info, but typically not in the middle of a chapter/section. make a note or call out in the text pointing to a section on why/how you got to the 'correct' position. That info is often helpful info, but can disrupt the flow of the 'good' information.

Re: Common mistakes using Kubernetes

#59

Post by barrkel »

xrd wrote: post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

s/tact/tack/

Tact means skill in dealing with other people, particularly in sensitive situations.

Tack has to do with sailing boats into the wind, and crucially "changing tack" means changing direction.

Re: Common mistakes using Kubernetes

#60

Post by jeffbee »

otterley wrote: post #55
jeffbee wrote: post #52

Earlier quoted context omitted.

If you ask for 0.001 CPU share, you might get it. I would advise caution. You that pod gets scheduled on a node with another node that asks for 4 CPUs and 100MB of memory, it's not going to get any time.

It depends. If the second pod requests 4 CPUs, it doesn't necessarily mean that the first pod can't use all the CPUs in the uncontended case. A lot of this depends on policy and cooperation, which is true for any multitenant system. If the policy is that nobody requests CPU, then the behavior will be like an ordinary shared Linux server under load - the scheduler will manage it as fairly as possible. OTOH, if there a…

Most people would be thrilled to get anything close to 25% CPU util. I guess one of the big missing pieces fro Borg that hasn't landed in k8s is node resource estimation. If you have a functional estimator, setting requests and limits becomes a bit less critical.
Post reply on HN