41–50 of 105 posts
It displays a pod containing multiple containers (this is fine and normal) but then highlights some of those containers to be different services.
Unless you guys are running wildly different setups, or we’re talking about sidecar and init containers which is a whole rabbit hole unto itself, I put different services in different pods.
This is a good start. It misses an important thing about Kubernetes that is often missed: extensibility. Each and every thing within Kubernetes can be swapped out for something else, including the scheduler. There are Custom Resource Definitions that supports these extensions and operators. For example, there is no built-in autoscaler for nodes, but someone wrote one and you can add one in there. It uses a constraint…
Thanks for the detailed answer > extensibility This is something that irks me right now, but at my current knowledge level (~0) it didn't feel right to even mention: If Kubernetes pushes everything out (CSI, CNI, Cloud provider integration, LoadBalancer, ...), doesn't it become "just a control loop runner"? I'm sure there's value in that, but I can imagine that most people running clusters would make different choice…
Even the core components can be swapped out so that Kubernetes can be adapted. For example, there are several teams that wrote kubelet replacements. The built-in scheduler for kubernetes is only responsible for pod placement by contacting the kubelet on a node and changing its desired state. The kubelet is responsible for actual placement.
That meant people have created kubelets that could run wasm workloads as pods. This was how native Windows workloads were implemented. I remember correctly, there are also kublets that could run unikernels, microvms, etc. The design pattern of label selectors remains the same, so that same concept can be used to place those specialized workloads.
I also learned something from an interview I once had ... their platform team came to the conclusion that every site is unique because every company will have its own specific needs and unique combination of technology. They (deliberately) don't use Kubernetes. They even crafted their interview process for that. It's a timed session to attempt to do something deceptively simple, that turned out to be an ops nightmare. There are no internet resources describing how to do that. No one is expected to finish it within the time. Instead, they are looking at how someone work through the problem as well as their emotional regulation while under time pressure.
Finally, I'd like to draw attention to a fallacy that has plauged modernity, and it is very applicable to computing design. This is from the field of political science, from a book written by James C. Scott called "Seeing Like a State". There is a blog post about the key concept -- legibility -- https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-call...
The pursuit of legibility has lead us to imposing a overly simplistic view onto something complex instead of really understanding the complexity. In the case of governments, this imposition uses authoritarian powers, and results of fragile solutions. In the case of computing infrastructure, we create abstractions and are surprised when it leaks.
> Why are we generating a structured language (YAML), with a computer, by manually adding spaces to make the syntax valid? Yep, it sucks. It's not like nobody has tried to do better, but nothing else has the adoption of Helm. Ultimately text is, as always, universal. If you want a fun fact: the communication between kubectl and the kube-api-server is actually in JSON, not YAML.
YAML is a super set of JSON. All JSON is valid YAML. No one uses YAML over the wire as you cannot guarantee the conversion from YAML to JSON as YAML is a super set and make contain things like anchors, which the JSON parser cannot handle.
That's why :)
The helm yaml thing really is annoying. Unfortunately it feels like helm is too firmly embedded to unseat at this point.
Here's what I would like from a Kubernetes-like system. I have a collection of machines. I have a very simple file that defines A) what I want to run, B) how the pieces communicate to each other, and C) how I want it to scale. Make that work without me having to think about any of the underlying infrastructure. (If this sounds suspiciously similar to Heroku, there you go)
There are tools that can do that, and are not as complex as Kubernetes I once interviewed at a place where their notion of "scaling" is to turn up the knob on Heroku. They also pay for it. They did not have anyone on their team who knew what questions to ask. They had grown to the point where they had outgrown their approach, and yet, have never developed the in-house capability. I mentioned the Cynefine framework el…
His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…
My take on the k8s yaml mess is, that we are lacking a modern replacement for Helm. The yaml templating on text file level is just pure crazyness. I think we would need an approach that is more like React/JSX or even better TSX (typescript JSX). Some possibility to get away from simple templating and more into structured descriptions, with typing and control flows.
I think there are some approaches into this direction (like the k8s Terraform module, or the defunct ksonnet), but none of them got it perfectly right yet.
The helm yaml thing really is annoying. Unfortunately it feels like helm is too firmly embedded to unseat at this point.
I've switched to kustomize for my homelab. Its built into kubectl (no extra tool needed) and its not really templating. It doesn't fully replace helm for 3rd party tools or packaging but its a solid alternative for less complicated setups.
His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…
> I've found solace in CUE We rebuilt our Kustomization layers to be built via cue, so that we could theoretically work with less mistakes. Oh my god, never again. Incredibly confusing and horrible errors. Writing new modules was painful and unclear, trying to compose them together was even worse. I still get flashbacks to that “Invalid bytes” error.
Like, people are correct that it is not great to allow (restricted at runtime or not) arbitrary code to define configs. I think "our config language is Python"-type statements are concerning in a whole lot of contexts, and for good reasons.
But holy shit at what cost. The landscape formed by the extremity of that no-real-scripting-languages-allowed sentiment is bleak. We've all dealt with it: the proliferation of handfuls of arcane tools that have to be present at specific versions in order to render a string. The triple-templated hellscapes written in Helm/Jinja/CUE/Jsonnet/DTL/ytt/JSON-patch/Dhall. Quoting hell. Indentation hell. Brace hell. The endless, endless bikesheds over which data language to render to (which are never really resolved and always just end up with JSON-interpolated-in-YAML-interpolated-in-TOML and transformed to the "right" format at render time). The flavor-of-the-month XKCD#927 creation of each iteration of a data+templating language that will solve all of the problems, but for real this time.
This really sucks. Not because its tedious; tedium is not necessarily a bad thing, with apologies to the automate-all-the-things crowd. But because it's expensive: in time, training, bugs, incident MTTR, and so on.
I dunno. Maybe we should accept that past some complexity threshold, you should just write your configs in a general purpose scripting language. Not because it's a good practice, but because our best attempts at an alternative are so much worse that it's probably the best practice to just do it in fucking Python. Starting from there and adding restrictions (i.e. Starlark) can't possibly be worse than the currently blessed way.
This was a useful read and somewhat gels with my experiences. Looking at the statement at the beginning > and it only requires you to package your workload as a Docker image, which seems like a reasonable price to pay. is no longer true as you continue down the path, since it actually requires you to do a lot more than you'd think.
I've not yet gone down that path, in which other ways does your workload need to adapt? I understand that you may want to get more value out of logs, and include side-cars, you may want inter-pod encryption and include service meshes, etc; but that'd be something that requires extra complexity in any setup
And yes, you're right, all these things are complex in any situation, except when you simply use a load balancer and two servers. There are companies estimated to be worth close to US$2B using with sub 30 servers using a service you may have heard of: Stack Overflow (https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha... && https://nickcraver.com/blog/2016/05/03/stack-overflow-how-we...).
(As an aside: K8s does not free you from your Cloud vendor - after floating around my city working on various "platforms", all locked in due to IAM, for example.)
Why is he mad at Kubernetes about Helm? Yes, Helm is a mess. Yes, lots of people use Helm to deploy things to Kubernetes, but critiques of Helm are not valid critiques of Kubernetes.