Kubernetes and back – Why I don't run distributed systems

Well, it depends on model how you manage servers. Cattle vs Pets. And of course true for takeover, thats nightmare whatever it have uptime or not ;)

I have just several servers here and there so their are all pets to me. Im 99% sure, server would boot cleanly. The remaining 1% is HW failure, with, can happen.

Example: I have a software FW running on some older HW here w/ 2 disks in Soft RAID 1. I had to power it down for 2 days due to electric maintenance in building. What a suprise was booting it. One disk failed right away making creepy noise. Other was ok, but server was booting long, so I hooked up monitor to check things out. Well, second disk was NOT in good condition either, it started to spit I/O errors here and there. Anyway. I managed to boot FW to working state, but immediatly got spare HW and started preparing for replacement. Now it runs 3 disks in Soft RAID 1 :)

> Clear signal that you can't justify the overheads.

There's a difference between an EC2 instance and EKS pricing.

Bare minimum, one control plane, zero usage, no workload, EKS starts at $873 per year.

EC2 starts at $36/year. t4g.small (2 vCPU, 2GB) is $145/year.

I run kubernetes as... a standalone node. I run it on a dedicated server with two disks on btrfs RAID1, with a subvolume for each pod that needs it (hostPath).

Not the minikube version, not the kubespray version: the kubeadm version, installed by hand and minimally tweaked (NodePorts 0-65535), with dual-stack networking support.

I have ~3 tenants, 5-15 pods each, and managing this configuration with docker was a steadily increasing pain for the last few years:

- I had docker-compose split-files for each service for each tenant to keep a consistent state, so I used a simple bash script to generate a really long docker-compose line to bring everything together, networking was a real pain

- Dual-stack support in docker does not exist/work in compose v3 to this day, but v2 has it

- Querying the configuration and state of a container was an exercise in debugging the output

- For certificates, I had to run letsencrypt/certbot externally (2 tenants with separate accounts for wildcard certs), but use traefik in docker-compose (the syntax pain was on top of the rest, of course)

Kubernetes simplified all the pain points above:

- I still have the service split-files, but I don't care about network addresses and links between pods anymore, just their names

- better isolation between all the architecture points, and clearer definition and integration

- IPv4&IPv6 native just working

- "describe" gets me all the information I need in a pretty format, "get type name -o yaml" if I need the configuration

- standard nginx ingress + cert-manager, and I can have multiple tenants without resorting to hacks

DBs are regular containers in the pod that requires them, just like any other container. Daily subvolume snapshots are small and effective to make sure the data is safe.

Kubernetes is a much better orchestrator than docker compose, availability is a plus if you really need it. You don't need to build an HA control plane plus 3 workers if you have no use for it.

Ah, unpatched and full of security vulnerabilities.

Oh I really like those comments. People think that keeping autoupgrades on without control keeps them secure. Right... And then you keep hearing stories about server wont go up because of update, or stuff breaks horribly. Not to mention brining new bugs to the table.

Its not that simple. I care what I install on my servers. Everything is carefully selected. Additionally, I follow KISS concept, so I run simple things that are managable.