A skeptic's first contact with Kubernetes

For example, there is no built-in autoscaler for nodes, but someone wrote one and you can add one in there. It uses a constraint solver for determining whether to expand or shrink node groups. If you want to use something else, you can find something or write it and install it.

Another example, you don't have to use kube-proxy. There are other ways to manage inter-node networking.

To address some of the questions:

> Why does it not matter if the state is unstable? If I'm operating a cluster that can't settle, I'd like to know immediately!

I'd point to the Cynefine framework to help make sense of this. Kubernetes is a tool that helps manage things in the complex domain, rather than the complicated domain. Unlike complicated systems, complex systems and complex adaptive systems may never reach a defined settled state.

> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"

That's already available. In addition to scaling on built-in metrics such as cpu and mem, there are ways to create custom metrics, including queue depth. You can do this because Kubernetes is extensible

> Why are the storage and networking implementations "out of tree" (CNI / CSI)?

It used to be in-tree, until the number of third party cloud and in-house storage and networking providers became unwieldy. This goes along with that fundemental characteristic of the Kubernetes design -- extensibility. AWS owns the Elastic Block Storage CSI, and GCP owns its CSI for its storage device. CNI allowed for the various service meshes, including exciting new ones such as the one based on eBFP.

The Cynefine framework again, helps sheds some light on this: the best way to respond to things in the complex domain is to try a lot of different approaches. Even if one approach doesn't look like it works now, some future state may make that previously impractical approach to work well.

> Given the above question, why is there explicit support for Cloud providers?

The current versions of Kubernetes pushes those implementations to CNI and operators. So for example, in order to make use of AWS ELB/ALB for the Ingress object, you have to additionally install the AWS-specific driver. If you are using AWS's EKS service, this is managed as an EKS addon. Under the hood, these drivers are, guess what, pods managed by replicasets managed by deployments that listens to the Kubernetes API servers for changes to the Ingress resource.

Not everyone uses it. On one of the Kubernetes sites I worked on, we used Traefik inside Kubernetes and its custom resource definition, IngressRoute. Everytime you create an Ingress, the AWS driver will create a completely new Load Balancer, which of course, drives up cost for very little gain.

I have a collection of machines.

I have a very simple file that defines A) what I want to run, B) how the pieces communicate to each other, and C) how I want it to scale.

Make that work without me having to think about any of the underlying infrastructure. (If this sounds suspiciously similar to Heroku, there you go)

I once interviewed at a place where their notion of "scaling" is to turn up the knob on Heroku. They also pay for it. They did not have anyone on their team who knew what questions to ask. They had grown to the point where they had outgrown their approach, and yet, have never developed the in-house capability.

I mentioned the Cynefine framework elsewhere, and I'll mention it again. Infrastructure and how it supports application, users, stakeholders, is a complex system.

A lot of people treat this as if it were in the complicated, or even clear domain. For certain class of use-cases, you can get away with this. But a company itself is a complex adaptive system, whose needs changes as it matures and scales ... or it doesn't, and it collapses.

The Cynefine framework describes four different domains, and the way you make decisions are different for each of them. You start getting in trouble when you attempt to use the decision-making process for one domain to a different domain. This often happens when the situation has evolved without the decision-makers noticing. It is easier for experts to see how clear domains changes to complicated domains, but it is more difficult to see when complicated domains change to complex domains.

Kubernetes is a sophisticated, adaptive, extensible tool for the complex domain. If it seems overly complex, it is because the domain is complex, and is not necessary if you are working from the complicated or clear domain.