Kubernetes YAML Generator

I don't think better abstraction is going to help the underlying problem: there are a lot of questions to answer, and you don't know the answer.

Without going into the complexity of Deployments, consider the lowly Pod. What configuration does your app need? What is the name of the container that contains it? How much memory does it use? How much CPU does it need? What ports does it listen on? What HTTP endpoint handles the health check? Does that endpoint test liveness or readiness? What filesystems does it need? What setup needs to be done before the main container runs? Does it need any special resources like GPUs? The list goes on.

The problem here is that when you're writing a Pod spec, you're building a single-purpose computer from scratch. In the traditional UNIX world, people answered most of these questions for you. How much RAM can my app use? However much I plugged in. How much CPU can my app use? All of them. What ports does it listen on? Any of them from 1024-65535. What filesystems does it need? Whichever ones I setup in /etc/fstab.

I don't think it's a stretch to call UNIX's "yolo" approach problematic. It is great when you have one server running one app, but servers have gotten gigantic (with pricing to match) while applications have largely stayed the same size. This means you have to pack multiple apps onto one physical server, and to do that, there have to be rules. When you write a Kubernetes manifest, you are just answering every possible question upfront so that the entire system runs smoothly even if your individual component doesn't. It's the cost of having small apps on big computers.

The problem comes from applications that you didn't write, or don't fully understand. Before you can understand how the application behaves, you have to write a manifest. But you don't know the answers to the questions like how much CPU you're going to use, or what the worst case memory usage is, etc. This causes a lot of cognitive dissonance, because the entire file is you admitting to the computer that you have no idea how to configure it. No abstraction layer is going to fix that problem, except by hiding those uncomfortable details from you. (And you will always regret using the "yolo" defaults -- who hasn't tried to SSH into a broken server only to have Linux helpfully OOMKill sshd or your bash instance when you're just trying to kill your malfunctioning app.)

This is largely the fault of application developers. They aren't willing to commit to reasonable resource limits because they don't want to handle support requests that are related to underprovisioning. My experience is that applications that set limits pick them wrong. For example, GCP and DigitalOcean's managed Kubernetes offerings both install monitoring agents to support their dashboards; these apps ship with limits that are too low and any reasonable Prometheus installation will notice that they are being CPU throttled and warn you about it. Now you have to waste your day asking "is this a real problem?"

Many open-source apps go the other way and pick resource limits that truly encapsulate the worst case and require individual nodes that are many times larger than the entire cluster. Yes, it would be nice if I gave each pod 32 CPUs and 128GiB of RAM... but I don't want to pay $2000/month/replica thankyouverymuch. (I've been on the other side of that where resources didn't cost me real money and happily used terabytes of RAM as cache.)

Application-level configuration is also not in a great state. Everyone tries to sell you their curated defaults so they don't have to write any documentation beyond a "quick start". (I'm as guilty of that as anyone in fact!) The application will have some built-in defaults (so the developers writing the app can just "go run main.go" and get the config they need). Then someone comes along to make a Helm chart for you, and they change the defaults so that their local installation doesn't need any customization. This only causes problems because instead of an undocumented underlying application, now you have that AND an undocumented abstraction layer. You may find the answer to your question "how do I configure FooApp to bar?" but have no way of communicating that config through the Helm abstraction layer because the author of the Helm chart never thought anyone would do that.

This rant has gotten quite long so I'll wrap it up. No abstraction layer is ever going to make it so you don't need to answer difficult questions. The actual list of questions to answer is available through "kubectl explain pod.spec" and friends, however.

Would you happen to have any links to Github repos that would be an example of this?

From light googling I only see examples of building gRPC APIs and microservices.

I would say kubernetes is very simple, in the same way lambda calculus or composing programs out of pipes is simple.

It has very simple basic model, which allows you to recursively built more and more complex abstractions on top of it.

I'm still building so I currently don't. Kubernetes is just a tool I use to deploy my customers sites on, it's not the selling point.

I do however have plans of supporting enterprise by packaging a version of the app into an on-prem deployment system so you can just deploy your own kubernetes cluster and install the app and have your own enterprise version of Primcloud.

Lots of PaaS offerings have been doing this for a long time, RedHat OpenShift being the original one; they contributed a lot of 1.0 k8s code and provided a lot of key early architectural feedback by implementing a PaaS using k8s. Another early one was Deis, who were at the first KubeCon.

All this is to say that you're at very least certainly half-correct, in that k8s is a very flexible tool that can be used to build a very simple, elegant, and ergonomic PaaS.

I'm not sure I agree that it's inappropriate for most businesses though, unless you think that only a PaaS like Heroku is appropriate for most businesses; the analogy I'd suggest is "Heroku vs. running your own VMs" circa 2010. Heroku is great for getting started, and lets you move fast by abstracting away a bunch of infra. But it's also restrictive; you can't pick and choose your components freely. As you grow past a certain point you'll almost certainly need the flexibility (or just cost-effectiveness) that you get from running your own infrastructure.

K8s is an improvement here because you can run a managed cluster on something like GKE, which takes away most of the operational toil, while still giving you a lot of flexibility on what components / pieces to include. The k8s domain API does a great job of abstracting away true infra concerns like volumes, compute, scheduling, load-balancing, etc, while making it really easy to package, use, and iterate on the stuff that sits on top of that infrastructure.

I'd probably not encourage a seed-stage startup to use k8s unless you're very familiar with the tool; a PaaS like Heroku would likely be more appropriate. However at the point that you'd usually graduate to running your own VMs (wherever that is on your own company's developmental path), I'd say that using k8s is now a better choice.