Is K8s Too Complicated? 

However I also think for a great deal of “every day” use cases solutions like GKE, EKS, Rancher, Cloud 66,... are good enough.

I had a ‘fun’ time figuring out the basics of things like ingress controllers in a bare metal setup because of this mishmash of docs.

Further, as it builds out standard abstractions for external services, it should enable applications to be written agnostic as to the specific cloud provider. That's pretty valuable. It has the potential to reduce AWS's lock-in advantage.

I agree people are running k8s HA in production, but there is a reason those people are dedicated k8s engineers. It's because it's a giant pain the ass to keep it running. Hence what I mean when I say it's "operationally complex".

Most people using GKE don't actually operate the k8s cluster, they let GKE run it for them. They just use it.

Using k8s and using nomad are similar from a developer perspective. Operationally they are night and day different.

Anyways, I suggest you go play with both systems, and try them out, put some non-important stuff in production under both of them.

Just imagine the complexity of something like APT on Debian/Ubuntu, or RPM on Red Hat/Centos. You could run into a problem with installing a package with apt-get or yum, perhaps some configuration script written in Bash that misbehaves during installation. To fix it, you have to understand how it's put together. The same applies to Kubernetes. You have to know the layers in order to work with them. Someone who doesn't know shell scripts or how init scripts work will not be able to work on Unix. Kubernetes is kind of like an operating system in the sense that it's a self-contained abstraction over something lower-level; the complexity of Unix isn't different, it's just that the design and implementation different.

Helm "just" installs parameterized YAML manifests. But Helm doesn't pretend to be an abstraction that simplifies Kubernetes. It simplifies the chore of interacting with Kubernetes, but in order to really use Helm, you have to understand what it is doing. Specifically, you do have to understand the "1000+ lines" of ingress declaration that it spits out. The notion that you can get around the complexity of Kubernetes with Helm is simply false.

To start with Kubernetes, take a step back, forget about Helm, and simply use Kubectl. You can accomplish everything absolutely you need with "kubectl apply -f". Learn each basic building block and how all of them fit together. Learn about pods before you learn about anything else. Deployments build on pods and are the next step. Then learn about services, configmaps and secrets. These are all the primitives you need to run stuff.

Ingresses are arguably the worst part of Kubernetes, since it's a pure declarative abstraction — unlike pods, for example, an ingress doesn't say anything about how to serve the ingress, it just expresses the end goal (i.e. that some paths on some hosts should be handled by some services). Ingress controllers are probably mysterious to beginners because they're an example of a "factory" type object: An ingress controller will read an ingress and then orchestrate the necessary wiring to achieve the end goal of the ingress.

Moreover, you don't need ingresses. Ingresses were invented a little prematurely (in my opinion) as a convenience to map services to HTTP endpoints have make these settings portable across clouds, but what most people don't tell you is that you can just run a web server with proxying capabilities, such as Nginx. This gist [1], which can be applied with "kubectl apply -f nginx.yml", describes a Nginx pod that will forward /service1 and /service2 to two services named service1 and service2, and will respond on a node port within the cluster (use "kubectl describe endpoints nginx" to see the IP and port). Assuming a vanilla Kubernetes install, it will work.

[1] https://gist.github.com/atombender/af2710818af0921e5c55a9ecb...