Live data from Hacker News

Common mistakes using Kubernetes

blog.pipetail.io

21–30 of 149 posts

Re: Common mistakes using Kubernetes

#21
post #9

Earlier quoted context omitted.

It’s an important point for those who may unknowingly over engineer.

And they are probably not going to change their mind by reading an anonymous sarcastic comment on HN.

For someone with a short time scale, only trawling this thread, of course not.

For someone young in this space, this comment and one hundred others -for and against- sift into hiso'er consciousness as part of the perceived zeitgeist of kubernetes within the larger community. Perhaps after several years, this person may have an intuition to avoid kubernetes in favor of separate docker or lxd containers.

That association of kubernetes as a FAANG-level tool builds stronger with the linked article: this hypothetical person can compare the struggles against perceived resources and so on. But not everyone has time to read the article, any given kubernetes article, that we come across. Some of those times, it's enough to take the temperature and move on. So, -over years- that may build an aversion that would not have otherwise formed had commenters avoided denouncing (or endorsing) kubernetes with less than full commitment toward convincing others.

Also, in this toneless medium, I can't intuit much of the emotional weight lolkube conveyed the sentence with. Was this person rueful, playful ?

Re: Common mistakes using Kubernetes

#22
post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another.

Your editor was very fucking wrong.

Re: Common mistakes using Kubernetes

#24

In my opinion, the most common mistake is not in the article : using kubernetes when you don't need to. Kubernetes has a lot of pros or the papers but in practice it's not worth it for most small and medium companies.

Do you also include managed kubernetes offerings, such as from Digital Ocean, in that assessment?

Re: Common mistakes using Kubernetes

#25
post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

So, so wrong. How did (s)he think experts get so good? Isn't the phrase `the master has failed more times than the apprentice has even tried` well known for a reason?

Re: Common mistakes using Kubernetes

#26

In my opinion, the most common mistake is not in the article : using kubernetes when you don't need to. Kubernetes has a lot of pros or the papers but in practice it's not worth it for most small and medium companies.

Do you also include managed kubernetes offerings, such as from Digital Ocean, in that assessment?

Not the OP but yes if cost is a factor. As far as I know no managed K8S offerings are cheap.

Re: Common mistakes using Kubernetes

#27
I think there is more to the story for some of these points and it can be dangerous to just take this at face value of best practices.

For example on the liveness / readiness probe item, the article says,

> “ The other one is to tell if during a pod's life the pod becomes too hot handling too much traffic (or an expensive computation) so that we don't send her more work to do and let her cool down, then the readiness probe succeeds and we start sending in more traffic again.”

But this is often a very bad idea and masks long term errors in underprovisioning a service.

If the contention of readiness / liveness checks vs real traffic is ever resulting in congestion, you need the failure of the checks to surface it so you can increase resources. If you set things up so this failure won’t surface, like allowing the readiness check to take that pod out of service until the congestion subsides, you’re only hurting yourself by masking the issue. It basically means your readiness check is like a latency exception handler outside the application, very bad idea.

The other item that is way more complicated than it seems is the issue about IAM roles / service accounts instead of single shared credentials.

In cases where your company has an enterprise security team that creates extremely low-friction tools to generate service account credentials and inject them, then sure, I would agree it’s a best practice to ruthlessly split the credentialing of every application to a shared resource, so you can isolate access and revoking.

But if you are on some application team and your company doesn’t have a mature enough security tooling setup managed by a separate security team, this can become a bad idea.

It can lead to superlinear growth in secrets management as there will be manual service account creation and credential propagation overhead for every separate application. Non-security engineers will store things in a password manager, copy/paste into some CI/CD tool, embed credentials as ENV permanently in a container, etc., all because they can’t create and maintain the end to end service account credential tools in addition to their job as an application team engineer. It’s something they think about twice per year and need off their plate immediately to move on to other work.

Across teams it means you end up with 20 different team-specific ways to cope with rapid growth of service accounts, leading to an even worse security surface area, risk of credential-based outages, omission of important testing because ensuring ability to impersonate the right service account at the right place is too hard, etc.

Very often it is a real trade-off to consider that one single service account credential that has just one way to be injected for every service is safer in the bigger picture.

Yes it means a credential issue for any service becomes an issue for all, and this is a risk and you want automated tooling to mitigate it, but it very often will be less of a risk than insisting on a parochial best practice of individual service account credentials, resulting in much worse and less auditable secrets workflows overall unless it is completely owned and operated by a central security team in such a way that it doesn’t create any approval delays or workflow friction for application teams.

Re: Common mistakes using Kubernetes

#28
post #6

Great article, I've learned many of these firsthand and agree with their conclusions. I have some more reading to do on PDBs! K8s is a powerful and complex tool that should only be used when needed. IMO you should be wary of using it if you're trying to host less than a dozen applications - unless it's for learning/testing purposes. It's a complex beast with many footguns for the uninitiated. For those with the right…

I'm not necessarily agreeing with you. Kubernetes really is a complex beast, and I wouldn't recommend self-hosting it for companies that don't have people that can focus solely on managing it.

I would also not recommend it for hosting a WordPress site, or a simple CRUD app.

However, when you get to the level where autoscaling is required, and where you are deploying multiple services, managed Kubernetes is not such a bad idea.

Using EKS (especially with Fargate) on AWS is not much harder than figuring out and properly utilizing EC2/ASG/ELB. GKE or DigitalOcean offerings seem to be even easier to use and understand.

Re: Common mistakes using Kubernetes

#29
I'm glad readers are liking the article, but please read and follow the site guidelines. Note this one: If the title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."

The submitted title was "10 most common mistakes using kubernetes", HN's software correctly chopped off the "10", but then you added it back. Submitters are welcome to edit titles when the software gets things wrong, but not to reverse things it got right.

Re: Common mistakes using Kubernetes

#30
post #6

Great article, I've learned many of these firsthand and agree with their conclusions. I have some more reading to do on PDBs! K8s is a powerful and complex tool that should only be used when needed. IMO you should be wary of using it if you're trying to host less than a dozen applications - unless it's for learning/testing purposes. It's a complex beast with many footguns for the uninitiated. For those with the right…

What would you suggest as an alternative, simpler form for docker deploy, running and managing? Docker-compose?
Post reply on HN