What we learned after a year on Kubernetes

> Increased billing from cross-AZ traffic

Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying.

(with alb-ingress-controller you can change target-type to IP to have traffic go direct to pods that will actually back a given service instead of via kube-proxy on a random node, but the big fault there is that if alb-ingress-controller gets hung up on something and you deploy, you can send traffic to nowhere because the target group still has old IPs for pods that don't exist).

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster.

The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything.

What I'd like is to generate serialized yaml files ... you know ... with code. Using some typed programming language, to be able to build any abstractions required for the job, and get some checks, compiler errors, and even ability to have asserts, unit-tests and so on. Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me.

Two benefits:

- I already use Helm to install open source software; there are already so many Charts available. So that makes it easy to also use Helm for my own apps (I use it just for templating though)

- Helm templating language is more flexible, but that can also bite you since it can become complex. With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching)

It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection to them was through a node it considers "unhealthy". I wrote at tool[1] to handle this at Houseparty before alb-ingress-controller supported this mode, and it gained us a whole nine as measured by the ALB.

The same failure mode actually exists with instances, though they typically rotate much more slowly. Monitor your ALB-ingress-controller and treat it as critical infrastructure, because it is.

[1]https://github.com/GauntletWizard/targetgroupcontroller

> With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching)

Config maps, enough said. I have a CD pipeline that writes environment variables to files, Kustomize eats them up and creates a configmap, environment variables in the deployment reference the config map, done.

Kapp as a layer on top of Kustomize is showing some progress. Ultimately Kustomize is like every other resource bundled with k8s, it’s a building block for higher level tooling.