Dear friend, you have built a Kubernetes (2024)
macchaffee.com
Dear friend, you have built a Kubernetes (2024)
1–10 of 183 posts
Re: Dear friend, you have built a Kubernetes (2024)
#2IMO, Kubernetes isn't inevitable, and this seems to paint it as such.
K8s is well suited to dynamically scaling a SaaS product delivered over the web. When you get outside this scenario - for example, on-prem or single node "clusters" that are running K8s just for API compatibility, it seems like either overkill or a bad choice. Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
There are also folks who understand the innards of K8s very well that have legitimate criticisms of it - for example, this one from the MetalLB developer: https://blog.dave.tf/post/new-kubernetes/
Before you deploy something, actually understand what the pros/cons are, and what problem it was made to solve, and if your problem isn't at least mostly a match, keep looking.
Re: Dear friend, you have built a Kubernetes (2024)
#3Unless you’re in Erlang world (Elixir, Gleam..) and all that is already baked into OTP and the BEAM. You can go on holiday knowing it will be a while longer before you need to break out the pods (and at that scale, you will be able to afford a colleague or two to help you).
Re: Dear friend, you have built a Kubernetes (2024)
#4The saddest part about Kubernetes is… after you set it all up, you still need a hacky deploy.sh to sed in the image tag to deploy! And pretty soon you’re back to “my dear friend you have built a Helm”. And so the configuration clock continues ticking…
Re: Dear friend, you have built a Kubernetes (2024)
#5> I know you wanted to "choose boring tech" to just run some containers.
The people advocating for boring tech generally aren't interested in containers.
You can just run programs.
Re: Dear friend, you have built a Kubernetes (2024)
#6The saddest part about Kubernetes is… after you set it all up, you still need a hacky deploy.sh to sed in the image tag to deploy! And pretty soon you’re back to “my dear friend you have built a Helm”. And so the configuration clock continues ticking…
I have been using Kubernetes for 7 or 8 years now, and have nearly 100% stayed away from Helm.
Some Kustomize, a little bit of envsubst and we're good to go thank you very much.
Re: Dear friend, you have built a Kubernetes (2024)
#7Found this the same day I published this: https://github.com/oddur/yoink
Kubernetes was overkill (I do that all day, 5 days a week); Kamal was too restrictive, so I found myself rolling out Yoink. Just what I need from k8s, but simple enough I can point it to a baremetal machine on Hertzner that can easily run all my workloads.
Re: Dear friend, you have built a Kubernetes (2024)
#8[deleted]
Re: Dear friend, you have built a Kubernetes (2024)
#9I need clarifications.
I see docker as a way to avoid having a standard dev platform for everyone in the company so that the infra team don't have to worry about patch xyz for library abc, only run docker.
But, with all the effort put in place to coordinate docker, k8s and all the shebang, isn't it finally easier to force a platform and let it slowly evolve over time?
Is docker another technical tool that tries to solve a non-technical problem?
Re: Dear friend, you have built a Kubernetes (2024)
#10Criticisms of Kubernetes generally come from a few places:
- People who would prefer their way of doing this, whether that's deployments on VMs, or use some sort of simpler cloud provider.
I had the same opinion a few years ago, but have kind of come to like it, because I can cleanly deploy multiple applications on a cluster in a declarative fashion. I still don't buy the "everything on K8s", and my personal setup is to have a set of VMs bought from a infrastructure provider, setup a primary/replica database on two of them, and use the rest as Kubernetes nodes.
- People who run Kubernetes at larger scales and have had issues with them.
This usually needs some custom scaling work; the best way to work around this if you're managing your own infra[1] is to split the cluster into many small independent clusters, akin to "cellular deployments"[2]/"bulkhead pattern"[3]. Alternatively, if you are at the point where you have a 500+ node cluster, it may not be a bad idea to start using a hyperscaler's service as they have typically done some of the scaling work for you, typically in form of replacing etcd and the RPC layer through something more stable.
- People who need a deep level of orchestration
Examples of such use cases may be to run a CI system or a container service like fly.io; for such use cases, I agree that K8s is often overkill, as you need to keep the two datastores in sync and generate huge loads on the kube-apiserver and the cluster datastore in the process, and it might be often better to just bring up Firecracker MicroVMs or similar yourself.
Although, I should say that teams writing their first orchestration process almost always run to Kubernetes without realizing this pitfall, though I have learned to keep my mouth shut as I started a small religious war recently at my current workplace by raising this exact point.
[1] Notice how I don't say "on-prem", because the hyperscaler marketing teams would rather have you believe in two extremes of either using their service or running around in a datacenter with racks, whereas you can often get bog-standard VMs from Hetzner or Vultr or DigitalOcean and build around that.
[2] https://docs.aws.amazon.com/wellarchitected/latest/reducing-...
[3] https://learn.microsoft.com/en-us/azure/architecture/pattern...