A skeptic's first contact with Kubernetes • Hacker News Board

Re: A skeptic's first contact with Kubernetes

#31

Post by hosh » Thu, Aug 01, 2024, 9:03 PM UTC

hbogert wrote: post #7

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…

The better way is to use manifest generators with a Turing-complete language. I wrote such a tool in Ruby.

The problem is that it is written in Ruby. It's great for shops that already have Ruby expertise. Wherever I have implemented this, it's worked out great. I have my doubts about it in other shops.

The community went a different way -- starting with templated JSON (not even JSON transforms). It was the least common denominator.

This is an issue with the ecosystem rather than design flaws of Kubernetes itself.

Re: A skeptic's first contact with Kubernetes

#32

Post by candiddevmike » Thu, Aug 01, 2024, 9:04 PM UTC

hbogert wrote: post #7

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…

Ksonnet being deprecated was one of the worst things to happen with the ecosystem IMO. Some kind of Jsonnet tool should have been integrated with kubectl like kustomize.

Shameless plug for Etcha, a configuration management tool (that works with kube!) built around Jsonnet manifests: https://etcha.dev

Re: A skeptic's first contact with Kubernetes

#33

Post by deivid » Thu, Aug 01, 2024, 9:05 PM UTC

Atreiden wrote: post #16

Great writeup on the core fundamentals, saved this to share with engineers who are new to k8s and need a quick primer. Re: This piece - > Given the Controller pattern, why isn't there support for "Cloud Native" architectures? > I would like to have a ReplicaSet which scales the replicas based on some simple calculation for queue depth (eg: queue depth / 16 = # replicas) > Defining interfaces for these types of events…

Using custom/external metrics is exactly what I was looking for, thanks!

I think my misunderstanding comes from the distinction between "first party" sensors (CPU scaler) and "third party" sensors (via "external metrics").

Is there a reason for this distinction? Will the CPU scaler eventually be pushed out of k8s as well?

Re: A skeptic's first contact with Kubernetes

#34

Post by hitpointdrew » Thu, Aug 01, 2024, 9:07 PM UTC

cyberax wrote: post #15

My problem with K8s: the network abstraction layer just feels _wrong_. It's an attempt to replicate the old model of "hard exterior, gooey interior" model of corporate networks. I would very much prefer if K8s used public routable IPv6 for traffic delivery, and then simply provided an authenticated overlay on top of it.

> My problem with K8s: the network abstraction layer just feels _wrong_.

> I would very much prefer if K8s used public routable IPv6 for traffic delivery

shudder... nothing could feel more wrong to me than public routable IPv6, yuck.

Re: A skeptic's first contact with Kubernetes

#35

Post by AcerbicZero » Thu, Aug 01, 2024, 9:09 PM UTC

This was a solid write up; I've been using K8s (intermittently) for like, 5 years now, and I still spend an inordinate amount of time looking things up and trying to convert the nonsense naming conventions used to something understandable. I can think of 20 or so projects that would have run great on K8s, and I can think of 0 projects that were running on K8s, which worked well.

Eventually, seeing the wrong tool used for the wrong job time and time again I came around to seeing K8s as the latest iteration of time sharing on a mainframe, but this time with YAML, and lots of extra steps.

Re: A skeptic's first contact with Kubernetes

#36

Post by deivid » Thu, Aug 01, 2024, 9:10 PM UTC

hosh wrote: post #22

This is a good start. It misses an important thing about Kubernetes that is often missed: extensibility. Each and every thing within Kubernetes can be swapped out for something else, including the scheduler. There are Custom Resource Definitions that supports these extensions and operators. For example, there is no built-in autoscaler for nodes, but someone wrote one and you can add one in there. It uses a constraint…

Thanks for the detailed answer

> extensibility

This is something that irks me right now, but at my current knowledge level (~0) it didn't feel right to even mention:

If Kubernetes pushes everything out (CSI, CNI, Cloud provider integration, LoadBalancer, ...), doesn't it become "just a control loop runner"?

I'm sure there's value in that, but I can imagine that most people running clusters would make different choices for various options, landing you with _every cluster being unique_, which feels wrong somehow.

Re: A skeptic's first contact with Kubernetes

#37

Post by [deleted] » Thu, Aug 01, 2024, 9:11 PM UTC

hbogert wrote: post #7

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…

[deleted]

Re: A skeptic's first contact with Kubernetes

#38

Post by richardwhiuk » Thu, Aug 01, 2024, 9:12 PM UTC

deivid wrote: post #33

Atreiden wrote: post #16

Great writeup on the core fundamentals, saved this to share with engineers who are new to k8s and need a quick primer. Re: This piece - > Given the Controller pattern, why isn't there support for "Cloud Native" architectures? > I would like to have a ReplicaSet which scales the replicas based on some simple calculation for queue depth (eg: queue depth / 16 = # replicas) > Defining interfaces for these types of events…

Using custom/external metrics is exactly what I was looking for, thanks! I think my misunderstanding comes from the distinction between "first party" sensors (CPU scaler) and "third party" sensors (via "external metrics"). Is there a reason for this distinction? Will the CPU scaler eventually be pushed out of k8s as well?

Kubelet is already managing the CPU resource I guess from scheduling?

I don't think there's any real movement on moving CPU scaler out of HPA.

Re: A skeptic's first contact with Kubernetes

#39

Post by cybrexalpha » Thu, Aug 01, 2024, 9:21 PM UTC

> Why are we generating a structured language (YAML), with a computer, by manually adding spaces to make the syntax valid?

Yep, it sucks. It's not like nobody has tried to do better, but nothing else has the adoption of Helm. Ultimately text is, as always, universal.

If you want a fun fact: the communication between kubectl and the kube-api-server is actually in JSON, not YAML.

Re: A skeptic's first contact with Kubernetes

#40

Post by FridgeSeal » Thu, Aug 01, 2024, 9:24 PM UTC

hbogert wrote: post #7

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…

> I've found solace in CUE

We rebuilt our Kustomization layers to be built via cue, so that we could theoretically work with less mistakes. Oh my god, never again.

Incredibly confusing and horrible errors. Writing new modules was painful and unclear, trying to compose them together was even worse. I still get flashbacks to that “Invalid bytes” error.