Kubernetes is hard

> Does a product focused, non infra startup even need k8s? In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product.

> Or just app engine honestly.

As a former App Engine PM and PM of Cloud Run, this warms my heart to hear--I'm glad folks love to use it :)

It's been a few years since I've worked on these products, but they were designed to be composed and used by each other. Cloud Run provides an autoscaling container runtime with Knative compatible API; Cloud Build (and Container Registry) + Cloud Run = App Engine, App Engine + CloudEvents = Cloud Functions.

With a Knative compatible API, you can theoretically take those primitives and move them to your own K8s cluster, managed (GKE/EKS/AKS) or self-managed, giving folks a tremendous amount of flexibility down the line if/when they need it (hint: the majority of customers didn't, and the fully managed services were great).

People think it took less effort.

Right up until you needed to do one of the very many things k8s implements.

For example, in multiple previous employers, we had cronjobs: you just set up a cronjob on the server, I mean, really, how hard is that to do?

And that server was a single point of failure: we can't just spin up a second server running crond, obviously, as then the job runs twice. Something would need to provide some sort of locking, then the job would need to take advantage of that, we'd need the job to be idempotent … all of which, except the last, k8s does out of the box. (And it mostly forces your hand on the last.)

Need to reboot for security patches? We just didn't do that, unless it was something like Heartbleed where it was like "okay we have to". k8s permits me to evict workloads while obeying PDB — in previous orgs, "PDBs" (hell, we didn't even have a word to describe the concept) were just tribal knowledge known only by those of us who SRE'd enough stuff to know how each service worked, and what you needed to know to stop/restart it, and then do that times waaay too many VMs. With k8s, a daemonset can just handle things generically, and automatically.

Need to deploy? Pre-k8s, that was just bespoke scripts, e.g., in something like Ansible. If a replica failed to start after deployment, did the script cease deployment? Not the first time it brought everything down, it didn't: it had to grow that feature by learning the hard way. (Although I suppose you can decide that you don't need that readiness check in k8s, but it's at least a hell of a lot easier to get off the ground with.)

Need a new VM? What are the chances that the current one actually matches the Ansible, and wasn't snowflaked? (All it takes is one dev, and one point in time, doing one custom command!)

The list of operational things that k8s supports that are common amongst "I need to serve this, in production" things goes on.

The worse part of k8s thus far has been Azure's half-aaS'd version of it. I've been pretty satisfied with GKE, but I've only recently gotten to know it and I've not pushed it quite as hard as AKS yet. So we'll see.

> Kubernetes is complex and I think they are partially right

Kubernetes is a distributed centralized operating system which itself depends on a distributed decentralized database, and has a varying network topology, permissions system, plugins, scheduler, storage, and much more, depending on how & where it was built, and runs applications as independent containerized environments (often deeply dependent on Linux kernel features) which can all have their own base operating systems. All of which must be maintained, upgraded, patched, and secured, separately, and frequently.

Kubernetes is literally the most complex single system that almost anyone in the world will ever use. It is the Katamari Damacy of the cloud.

> It allows dev teams to not worry about all these things; all they must do is to write a simple YAML file.

cackles, then sobs

> More importantly, teams no longer need to ask DevOps/infra folks to add DNS entry and create a Load Balancer just to expose a service.

more sobbing

> Should you use Kubernetes?

Should you change a tire with a crowbar, a can of WD40, and a lighter? Given an alternative, the alternative is usually better, but sometimes you don't have an alternative.

Your company "saves" over 100k/month paying WAY too much for EKS, which is extremely expensive.

If you're at any decent scale (looks like you are), then switch to GKE, or switch to on-prem and buy some hardware + a Kubernetes distro like Mirantis/Openshift/Tanzu.

Heck, go run k3s on Hetzner and you won't have that much more work, but save literally millions at the scale you're talking about.

Serverless does not necessarily mean lambda. It could be just about anything that runs containers for you. AWS ECS has an offering called Fargate that I've been happy with for our hosting. You are right though that the compute costs are typically more than renting a traditional VPS. There is definitely a tradeoff between labor and compute costs.