Highly available Kubernetes with batteries for small business

For years I bemoaned that it doesn't scale down to the scale where most engineers work. I was wrong about that, it didn't scale down to the amount of learning I was willing to do for any one project at the scale I was doing them.

SOOO many times have I had to try to make my tools do something they weren't really capable of doing. And while it feels like a win today it never does tomorrow.

With Kubernetes most of my issues come from other things than that itself. Digital Ocean has a great small scale experience. AWS EKS has been quite a mess in comparison in terms of Ops time required. This made me realize that maybe people are suffering with ineffifcenies from the cloud provider, not Kubernetes itself.

2. "do CoLocos" definitely has effort and does not in anyway guarantee HA. (HA is really more of a software application layer concern not hardware) Also, are you telling me you are going to engineer an HA load balancer service better than the people who work for the major cloud vendors? Are you going to setup all your own service discovery, scheduling and everything else k8s does for your application?

3. Prometheus and other solutions are so much more than "/var/log + bash grep" you can't possibly seriously equate the two. Serious applications demand serious metric solutions.

4. Sure let me do all that while my site starts getting the HN hug of death or you know I could do nothing and let my infrastructure autoscale for me. Even in a non-autoscale setup, all you have to do is change a couple of numbers in your terraform and k8s config files and reapply.

5. No it isn't. For some webservers like Caddy it's easier than others, but in many cases you are going to have to a good amount of manual steps to get auto renewal working for your Let's Encrypt certs. On K8s it's basically a config file and a helm chart install. Plus you have K8s guaranteeing that it will restart the renewal service should it go down.

6. Yes you can specify config in your webserver for the routing, but being able to run completely distinct application servers on the same nodes handling all the routing and service resiliency takes a lot of effort, while not running into conflicts with your existing applications that are already running. You are going to get a lot better bang for your buck running multiple applications on the same set of nodes than having to provision separate nodes for the other applications.

7. SSH + cli does not give you a cluster wide view of your application. You would have to feed all that information into a single location to be able to view it, which would take a lot of effort and still wouldn't provide the same level of detail.

As a small business, I'd much rather have well engineered solutions that will be able to grow with my business. A lot of what you describe might be okay when your business is very small and has a single server (basically you can afford to treat it as a pet), but as your business grows you want to avoid having to rework various parts of your infrastructure if possible. Managed k8s gives you a lot of room to grow with relatively little upfront time investment.

It's about providing leverage to your team so you need less effort to do more. And SMB can be anywhere up to 1000 employees if using semi-legal definitions, 500 employees and under 1 billion euro of turnover in case of EU's SME definition.

Depending on the area of work, there can be a lot of software to run. And that's where k8s comes in. It might seem "hard" when you're comparing it to "just throw few packages on a server", but having gone through that recently? The next time I'm setting up a single-node server to support tools for a project, I'm using k3s instead of spending 3-5 days getting software to play nice with paths changed to non-standard locations in order to keep data easy to backup and migrate to another server. Or wrestling load balancer (and ours is IaC even! Still not as nice and easy to deal with as nginx-ingress-controller...).

The real "scaling" of kubernetes is that once you pass the initial hurdle, the cost and complexity of adding more applications is greatly lowered. Even when you do a dumb lift-and-shift (like I did for >60 applications in one project).