Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Common mistakes using Kubernetes

blog.pipetail.io

Post reply on HN
Print view View original

91–100 of 149 posts

Re: Common mistakes using Kubernetes

#91

Post by m463 »

xrd wrote: post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

years (and years) ago, if I wanted to learn a new computer technology or language, I would pick up a book and learn it.

One language that didn't go how I expected it was applescript.

The book on applescript (from o'reilly I believe) was necessarily different.

Instead writing "normal" top-down programs, applescript hooks into the OS from the side.

Many books can concentrate on what you can do, but the applescript had to do everything by example. This is because all of apple's applications expose interfaces that you have to figure out on the fly.

I had to learn a different way from that book, because it necessarily concentrated on "how" instead of "what".

Re: Common mistakes using Kubernetes

#92

Post by ghostpepper »

recursive wrote: post #85
snupples wrote: post #82

Earlier quoted context omitted.

Yes we need it. This is absolutely becoming a tiresome trope. K8s is a huge benefit to tons of companies and none of them are Google. Yes some people are using K8s when they don't need it. Just like many are using cloud managed services when they don't need them. Or vms. Or insert any technology here. This article has nothing to do with whether K8s fits some particular use case but may be of help (although I disagree…

Kubernetes doesn't benefit Google? How not?

I think the intention of the post you replied to was to say that many companies other than google have a legitimate need for kubernetes.

Re: Common mistakes using Kubernetes

#93

Post by dharmab »

garadox wrote: post #86

One thing to watch for with pod antiAffinity - if you use required vs preferred, and your pod count exceeds the node count, the remainder will be left in Pending and won't spin up anywhere.

There's a new feature which does a better job of spreading Pods without blocking scheduling quite as badly: https://kubernetes.io/docs/concepts/workloads/pods/pod-topol...

Re: Common mistakes using Kubernetes

#94

Post by dharmab »

yongjik wrote: post #79
tomnipotent wrote: post #12

Earlier quoted context omitted.

To be fair, I don't see how this could be shorter in any other language without losing readability.

You don't have to. Kubernetes should just say "required" here, and the documentation should say "Warning: this is only checked during scheduling." printf isn't named printIntoBufferWhichMayNotFlushUntilLinefeed, and people are fine with it.

I believe there may have been a plan to allow checks during runtime at some point; although this feature is no longer necessary since https://github.com/kubernetes-sigs/descheduler#removeduplica... can do it.

Re: Common mistakes using Kubernetes

#95

Post by bufferoverflow »

speedgoose wrote: post #19

In my opinion, the most common mistake is not in the article : using kubernetes when you don't need to. Kubernetes has a lot of pros or the papers but in practice it's not worth it for most small and medium companies.

Most of the time you don't even need a cluster of any kind. We live in a world where you can spin up a server with 3TB of RAM and 128 cores whenever you want, and it will cost you less than a senior developer.

Re: Common mistakes using Kubernetes

#96

Post by yongjik »

Well, nobody asked me, and I'm no expert, but here's my list of what (not) to do in Kubernetes (if I had the authority).

1. There. Is. No. Machine. (Insert matrix meme here.) Before you open up your cluster to the rest of company, drill it down to them. Maybe even create a Google Form where they have to sign "I hereby acknowledge that there is no machine in k8s and any attempt to tie my job to a particular machine means a broken config by definition."

2. Thanks to 1, don't let anyone use hostNetwork, hostIPC, hostPID, hostPorts, host whatever, unless you have a really good reason to (with explicit approval process).

3. Don't let anybody start a job without memory/CPU limit. Make sure they understand that, if the job goes over the memory limit, it dies, and it's not k8s admin's problem.

4. You can't log anything into the pod - when the pod dies the log is gone. You can't log into the machine, either (see 1). Therefore, you really need some kind of logging framework that takes the log from your pod and saves it, in its raw form, somewhere safe (like S3). I don't know if there's any such framework, but there had better be.

5. Make sure every manual operation is logged (who did what to which job when), unless you like asking "@here Does anybody know who owns fooservice?" every month.

6. Kubernetes is not magic: if it takes thirty minutes to provision your service, fix that, instead of moving thirty minutes of manual provisioning into k8s and somehow expect it to be magically reliable.

7. Don't bring in existing dependencies uncritically. If your job connects to a zookeeper server to find out its peers, don't bring it into k8s, but rewrite it to use k8s service instead.

8. Take extra extra care when writing down your first job specification, because there are a lot of yaml files to write, and people will just copy what's already there. If your first k8s job mounts host /tmp directory just because you were testing something and forgot to delete the line, soon you will have fifty jobs all mounting host /tmp directory. Good luck figuring out which job actually needs it then.

Yeah, again, I'm by no means an expert - I'm not even an admin, so just consider the list as a rambling of some poor soul who has seen some stuff. Here be dragons, have fun.

Re: Common mistakes using Kubernetes

#97

Post by chx »

tcbasche wrote: post #81
chx wrote: post #23

It misses the biggest one: using it. I ranted about the cloud a decade ago http://drupal4hu.com/node/305 and there's nothing new under the Sun. Still most companies doing cloud and Kubernetes doesn't need it... practice YAGNI ferociously.

I think you may be on the wrong side of history here

Nothing new with that one. I still think git was the wrong choice for DVCS and yet, I have been using it since for a decade or more now. I am still still feeling I have an uneasy truce with it but not a friendship. I still think github is a shitty choice for hosted git -- at least most large open source projects have went with gitlab so I am not utterly alone with that. I am using now Kubernetes because my primary client is using it. Doesn't mean I am happy with it or that I think it's necessary by any means. It's fine. I am getting old but I still can learn. Doesn't mean I can't be grumpy about it.

Re: Common mistakes using Kubernetes

#98

Post by drvdevd »

fnord123 wrote: post #22
xrd wrote: post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

Agreed. The entire expertise of programming is based on the willingness to find these sorts of issues and fix them. Over and over, for your entire career.

Re: Common mistakes using Kubernetes

#99

Post by drvdevd »

xrd wrote: post #71
fnord123 wrote: post #22

Earlier quoted context omitted.

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

Obviously, I agree with you! I also think this is the way the majority of tech books are written. Can you think of another where the author goes from mistake to mistake and then finally gets it right? I believe there is a space in tech writing for this kind of writing, but it is not something traditional book publishers believe. This was an O'Reilly book by the way, with really good editors and a really good editoral…

Most (in-depth, technical) security books follow this sort of thought process- I think. “A big hunters diary” comes to mind.

Re: Common mistakes using Kubernetes

#100

Post by hinkley »

jeffbee wrote: post #53
hinkley wrote: post #36

> You can't expect kubernetes scheduler to enforce anti-affinites for your pods. You have to define them explicitly. Why isn't this the default behavior? Why don't I have to go in and tell it that it's okay to have multiple instances on the same node? Why? So that I somehow feel like I've contributed to the whole process by fixing something that never should break in the first place? I know of a few pieces of code wh…

It's quite possible that you have a machine with 192 CPU cores in it, but it's very unlikely that you are able to write a service that scales to that level ... and if you write it in Go it's really unlikely that you can scale even to 8 CPUs. There's nothing weird about having multiple replicas of the same job on the same node. If you look through the Borg traces that Google recently published you can find lots of job…

This is not how defaults work.

When you are talking about the realm of the possible, you provide settings that allow you to reach the scenarios that you feel are reasonable, desirable, or lucrative (or commonly enough, some happy combination of the three).

Defaults are the realm of the probable. And nobody is requisitioning a 192 core machine without a good bit of due diligence, which would include deciding how to set server affinity.

Post reply on HN