A skeptic's first contact with Kubernetes
blog.davidv.dev
A skeptic's first contact with Kubernetes
1–10 of 105 posts
Re: A skeptic's first contact with Kubernetes
#2[deleted]
Re: A skeptic's first contact with Kubernetes
#3> Why are the storage and networking implementations "out of tree" (CNI / CSI)? Given the above question, why is there explicit support for Cloud providers?
eg: LoadBalancer supports AWS/GCP/Azure/..
Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree. The upcoming 1.31 release will drop a lot of existing, already deprecated support for AWS & others from Kubernetes proper. https://github.com/kubernetes/enhancements/blob/master/keps/...
There's some plan to make this non-dosruptive to users but I haven followed it closely (I don't use these providers anyhow).
> Why are we generating a structured language (YAML), with a computer, by manually adding spaces to make the syntax valid? There should be no intermediate text-template representation like this one.
Helm is indeed a wild world. It's also worth noting that Kubernetes is also pushing towards neutrality here; Helm has never been an official tool, but Kustomzie is builtin to kubectl & is being removed. https://github.com/orgs/kubernetes/projects/183/views/1?filt...
There's a variety of smart awesome options out there. First place I worked at that went to kube used jsonnet (which alas went unmaintained). Folks love CUE and Dhall and others. But to my knowledge there's no massive bases of packaged software like exists for Helm. Two examples, https://github.com/bitnami/charts/tree/main/bitnami https://github.com/onedr0p/home-ops . It'd be lovely to see more works outside Helm.
Thanks sysdig for your 1.31 write up, https://sysdig.com/blog/whats-new-kubernetes-1-31/
Re: A skeptic's first contact with Kubernetes
#4> Why are the storage and networking implementations "out of tree" (CNI / CSI)? Given the above question, why is there explicit support for Cloud providers? eg: LoadBalancer supports AWS/GCP/Azure/.. Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree. The upcoming 1.31 release will drop a lot of existing, already deprecated support for AWS & others from Kubernetes proper. http…
> Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree.
I wasn't aware, but it makes sense, and explains the current "confusing" state.
Re: A skeptic's first contact with Kubernetes
#5The helm yaml thing really is annoying. Unfortunately it feels like helm is too firmly embedded to unseat at this point.
Re: A skeptic's first contact with Kubernetes
#6I agree with:
> My opinion is that a large part of Kubernetes' value is derived from just two concepts
I agree with the first one, "control loops," but the second one is "API-based," not "Services."
Re: A skeptic's first contact with Kubernetes
#7His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets.
I've found solace in CUE which I just run locally to catch all the small errors everybody makes on a daily basis. Putting the CUE validation in our pipeline is too confronting for others, yet they're constantly making up best practices adhoc during reviews which could've easily been codified with CUE (or some other serious config language).
Re: A skeptic's first contact with Kubernetes
#8Nice article ;)
> The number of rules may be large, which may be problematic for certain traffic forwarding implementations (iptables linearly evaluates every rule)
Kube proxy also supports ipvs out of the box, and some CNI (like Cilium) can also replace kube proxy and rely on eBPF.
> When a Pod is moved to another node, the traffic will be forwarded twice until the old DNS entry expires Not sure to understand this one. On a standard setup what happens is:
- Pod A is running on a node, receiving traffic from a service
- The pod is stopped by kubelet (that send a SIGTERM to it).
- The pod should gracefully shutdown. During the shutdown phase, only _existing_ connections are forwarded to the stopping pod, new ones will be already forwarded elsewhere.
- If the pod stops before the terminationGracePeriodSeconds duration (default 30s), everything is fine. Else, the pod is killed by kubelet. So it's developers that should make sure pods handle signals correctly.
"Misbehaving clients (eg: ones that do not re-resolve DNS before reconnecting) will continue to work" => the services IP is stable so clients don't need to re-resolve.
> Why does it not matter if the state is unstable? If I'm operating a cluster that can't settle, I'd like to know immediately!
Kubernetes exposes a lot of metrics, on the control plane components or kubelet, usually using the Prometheus format. Look for example at the metrics exposed by kube state metrics: https://github.com/kubernetes/kube-state-metrics/tree/main/d...
With controllers metrics + kube state metrics about most Kubernetes resources, you can easily build alerts when a resource fails to reconcile.
> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"
Take a look at KEDA, it's exactly this: https://keda.sh/ It "extends" the autoscaler capabilities. If you're running Prometheus you can for example scale on any metric that is stored in Prometheus (and so exposed by your application/infrastructure components: queue depth, latency, request rate...).
Kubernetes was built to be extended like this. Same for your question "Why are the storage and networking implementations "out of tree" (CNI / CSI)?", to my experience support is very good today on various cloud providers or on premise infra components. Look at Karpenter for example, it's IMO a revolution in the Kubernetes node management world.
Re: A skeptic's first contact with Kubernetes
#9This was a useful read and somewhat gels with my experiences.
Looking at the statement at the beginning
> and it only requires you to package your workload as a Docker image, which seems like a reasonable price to pay.
is no longer true as you continue down the path, since it actually requires you to do a lot more than you'd think.
Re: A skeptic's first contact with Kubernetes
#10> Why are the storage and networking implementations "out of tree" (CNI / CSI)? Given the above question, why is there explicit support for Cloud providers? eg: LoadBalancer supports AWS/GCP/Azure/.. Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree. The upcoming 1.31 release will drop a lot of existing, already deprecated support for AWS & others from Kubernetes proper. http…
> Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree. I wasn't aware, but it makes sense, and explains the current "confusing" state.
Thanks for the write-up, and especially for storytelling your ability & willingness to wade in & find out! Proper hacker spirit!
You did such a great job offering an informative high level view (honing in on the control loop feels like seizing upon a critical insight), & iterating through bits nicely. You were tactful & clear in raising some ongoing qualms, which indeed seem largely correct & frustrating, even to those who started as believers. Thanks thanks, a recommendable article indeed.