Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A skeptic's first contact with Kubernetes

blog.davidv.dev

Post reply on HN
Print view View original

71–80 of 105 posts

Re: A skeptic's first contact with Kubernetes

#71

Post by lmm »

movedx wrote: post #49
deivid wrote: post #11

Earlier quoted context omitted.

I've not yet gone down that path, in which other ways does your workload need to adapt? I understand that you may want to get more value out of logs, and include side-cars, you may want inter-pod encryption and include service meshes, etc; but that'd be something that requires extra complexity in any setup

OIDC integration, RBAC, data persistence/replication across nodes, observability, mTLS. And yes, you're right, all these things are complex in any situation, except when you simply use a load balancer and two servers. There are companies estimated to be worth close to US$2B using with sub 30 servers using a service you may have heard of: Stack Overflow ( https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...…

> And yes, you're right, all these things are complex in any situation, except when you simply use a load balancer and two servers.

How does that simplify or solve any of the problems you mention? As far as I can see they're just as present and just as complex when you have two servers as when you have hundreds.

Re: A skeptic's first contact with Kubernetes

#72

Post by lmm »

jgalt212 wrote: post #61

What's the break-even number of machines you must manage before Kubernetes starts to make sense?

I'd say 3 or more distinct technology stacks. You can run 200-machine clusters comfortably with lighter tooling if they're all Ruby or all JVM or what have you (and I have), but once your deployment process requires remembering the foibles of 4 different Python build tools it becomes easier to shove it all in Kubernetes.

Re: A skeptic's first contact with Kubernetes

#73

Post by mst »

zbentley wrote: post #53
movedx wrote: post #43

Earlier quoted context omitted.

> If you want a fun fact: the communication between kubectl and the kube-api-server is actually in JSON, not YAML. YAML is a super set of JSON. All JSON is valid YAML. No one uses YAML over the wire as you cannot guarantee the conversion from YAML to JSON as YAML is a super set and make contain things like anchors, which the JSON parser cannot handle. That's why :)

Nobody tell the JSON::XS CPAN maintainer you said that: https://metacpan.org/pod/JSON::XS#JSON-and-YAML Or John Millikin: https://john-millikin.com/json-is-not-a-yaml-subset

One of many reasons why most of the perl devs I know use the Cpanel::JSON::XS fork now.

Re: A skeptic's first contact with Kubernetes

#74

Post by Jtsummers »

yen223 wrote: post #67
hbogert wrote: post #7

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…

Something curious I noticed: while this comment was posted "6 hours ago", when I click reply, it instead says that the comment was posted "3 days ago". Is this happening to others as well? From digging around it looks like the comment really was posted 3 days ago, looking at the `title="2024-07-29T08:41:07"` attribute in the timestamp tag.

Submissions get renewed sometimes and the time stamps are set to a newer time (this is to get it ranked high enough to hit the front page). All comments have a false (newer) time stamp for a while if they predate the renewed time stamp. After a few hours the original time is restored for the submission and comments.

These usually come from the second chance pool (click lists at the bottom to get to the pool and some other lists).

Re: A skeptic's first contact with Kubernetes

#75

Post by smarterclayton »

richardwhiuk wrote: post #38
deivid wrote: post #33

Earlier quoted context omitted.

Using custom/external metrics is exactly what I was looking for, thanks! I think my misunderstanding comes from the distinction between "first party" sensors (CPU scaler) and "third party" sensors (via "external metrics"). Is there a reason for this distinction? Will the CPU scaler eventually be pushed out of k8s as well?

Kubelet is already managing the CPU resource I guess from scheduling? I don't think there's any real movement on moving CPU scaler out of HPA.

CPU tracking is provided by the metrics API, which either reads kubelet metrics directly (the original, old, but simplest way), or a metrics adapter that reads the metrics from a third party collector and implements the API.

The behavior is supported by the v1 api rules so no, it’s extremely unlikely to be moved out.

That said, with GPU workloads gaining steam I wouldn’t be surprised if we added new “supported everywhere” metrics at some point.

Re: A skeptic's first contact with Kubernetes

#76

Post by hosh »

zbentley wrote: post #48
FridgeSeal wrote: post #40

Earlier quoted context omitted.

> I've found solace in CUE We rebuilt our Kustomization layers to be built via cue, so that we could theoretically work with less mistakes. Oh my god, never again. Incredibly confusing and horrible errors. Writing new modules was painful and unclear, trying to compose them together was even worse. I still get flashbacks to that “Invalid bytes” error.

I think it's worth re-examining the widespread opposition to writing configs in an interpreted "real" programming language, rather than a string templating language (if you're looking to split hairs, in this taxonomy PHP and JSX are real languages, but they're close to the line. Python is prototypically a real language. Awk, Jinja, CUE, and Bash/ZSH string expansion are not. If the word "real" bothers you, pick anoth…

I think that templated languages are the lowest common denominator.

Examples for using real languages for configuration management is Chef. One of the bigger hurdles is that someone with an ops background has to learn a language.

On the other hand, if you are recruiting someone for an SRE or platform team, you are looking for ops people that are willing to learn how to dev, or dev people willing to get their hands dirty with ops.

Re: A skeptic's first contact with Kubernetes

#77

Post by hosh »

andrewaylett wrote: post #55
zbentley wrote: post #48

Earlier quoted context omitted.

I think it's worth re-examining the widespread opposition to writing configs in an interpreted "real" programming language, rather than a string templating language (if you're looking to split hairs, in this taxonomy PHP and JSX are real languages, but they're close to the line. Python is prototypically a real language. Awk, Jinja, CUE, and Bash/ZSH string expansion are not. If the word "real" bothers you, pick anoth…

I don't object to the desired state being represented in YAML, I object to generating that YAML using something that's not a "real" programming language. I don't particularly care if you generate your data structures from schemas, or generate your schemas from data structures, but I do care if you don't generate your output YAML using anything that's not type-safe. If you're generating YAML to deploy, you can do that…

I generally agree.

I use Ruby for this. Duck typing is sufficient.

I’ve also found that coupling manifests generation with application deployment to close a lot of doors. It’s better to modify manifests to change image references (or something similar), and have a separate process for applying manifest changes.

Re: A skeptic's first contact with Kubernetes

#78

Post by whalesalad »

cyberax wrote: post #15

My problem with K8s: the network abstraction layer just feels _wrong_. It's an attempt to replicate the old model of "hard exterior, gooey interior" model of corporate networks. I would very much prefer if K8s used public routable IPv6 for traffic delivery, and then simply provided an authenticated overlay on top of it.

There are ways to unbolt the native networking stack and roll your own. Tons of options available: https://github.com/containernetworking/cni

I don’t agree with your approach (curious as to why you would want this) but I believe it’s possible.

Re: A skeptic's first contact with Kubernetes

#79

Post by liggitt »

mtndew4brkfst wrote: post #12
jauntywundrkind wrote: post #3

> Why are the storage and networking implementations "out of tree" (CNI / CSI)? Given the above question, why is there explicit support for Cloud providers? eg: LoadBalancer supports AWS/GCP/Azure/.. Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree. The upcoming 1.31 release will drop a lot of existing, already deprecated support for AWS & others from Kubernetes proper. http…

Kustomize is builtin to kubectl & is being removed Hadn't heard this until now, I'm a rather happy user after being jaded about Helm 2/3. Do you happen to know if this removal is sentimentally closer to "we don't want to keep maintaining kustomize" or to "we don't want to keep embedding kustomize, please use its binary directly"?

The motivation is more the latter, but it's not at all clear the proposed removal of the embedded kustomize will proceed, given the compatibility implications. See discussion at https://github.com/kubernetes/enhancements/issues/4706#issue... and following.

Re: A skeptic's first contact with Kubernetes

#80

Post by hosh »

lmm wrote: post #66
hosh wrote: post #31

Earlier quoted context omitted.

The better way is to use manifest generators with a Turing-complete language. I wrote such a tool in Ruby. The problem is that it is written in Ruby. It's great for shops that already have Ruby expertise. Wherever I have implemented this, it's worked out great. I have my doubts about it in other shops. The community went a different way -- starting with templated JSON (not even JSON transforms). It was the least comm…

Sort of. Kubernetes is lowest common denominator by design. Specific ecosystems had much more efficient solutions to these problems (e.g. rather than having to deploy a whole docker container to do a version upgrade you deploy just your specific web application) but the problem was those solutions only worked for those specific ecosystems.

I disagree. Kubernetes is composed from primitives that can be applied in versatile ways. Those primitives can be swapped out for even greater extensibility. Kubernetes is not the lowest common denominator, and more of a toolkit that can broadly and deeply applied to many setups.

Templated yaml or json is a step backwards. Those are not composable. If we wanted something like that, then we would need to at least be able to do merges and transforms. Something like jquery or css for yaml that allows to transform fields based upon selectors.

Manifest generators using a real language, on the other hand, start with data structures that can be merged and transformed before its final output as yaml. They can take advantage of language features. The one I wrote in Ruby can take advantage of class inheritance and mixins, allowing me to define common configs across resources. Someone who can distill the algebra of merging Kubernetes manifests can do that for a functional programming language.

Post reply on HN