A skeptic's first contact with Kubernetes • Hacker News Board

Re: A skeptic's first contact with Kubernetes

#101

Post by dilyevsky » Sat, Aug 03, 2024, 6:00 AM UTC

cyberax wrote: post #100

dilyevsky wrote: post #99

Earlier quoted context omitted.

So basically i need to run another piece of infra that does NAT64 and DNS64 and limits my deployment options quite a bit (can't do DSR)? Totally unnecessary in cloud... Not sure how that's better for users but probably better for vendors ;) Btw, overlay is not the only option to do CNI - Calico, Cilium and few others can do it via l3 by integrating with your equipment. Even possible in cloud but has serious scale lim…

No, you misunderstand me. My dream infrastructure would run IPv6 with publicly routable IP addresses for the internal network, for everything. IPv4 is needed only for the external IPv4 clients, and for the server code to reach any external resources that are IPv4-only. The clients are simply going to connect via the border load balancers, just as usual. For the external IPv4-only resources, you'll need to use DNS64.…

> It's still an overlay network, just in hardware.

It really isn't, at least not in commonly understood sense. See [0] for example - you can use this with dual-stack and route everything natively even with ipv4 using rfc1918 cidrs. No ipip/gre/vxlan tunneling required. Does require setting up BGP peering on your routers.

[0] - https://cloudnativelabs.github.io/post/2017-05-22-kube-pod-n...

Re: A skeptic's first contact with Kubernetes

#102

Post by hosh » Sat, Aug 03, 2024, 4:27 PM UTC

FridgeSeal wrote: post #94

hosh wrote: post #77

Earlier quoted context omitted.

I generally agree. I use Ruby for this. Duck typing is sufficient. I’ve also found that coupling manifests generation with application deployment to close a lot of doors. It’s better to modify manifests to change image references (or something similar), and have a separate process for applying manifest changes.

I’ve had great success with Flux and a small bit of CI for this. We currently have configs in raw K8s yaml in our monorepo in kustomize base+overlay style setup. It’s super duper obvious what’s in each env, and straightforward to change them. Images are swapped in using Kustomize’s “newImage” functionality. Flux watches git and deploys to K8s. At any point it’s super clear what the configs are, what image we were usi…

Thanks for sharing that. That validates my observation. I had implemented a system with separate manifest and deployment and it had worked great. I came to a team with an existing process that coupled manifest and deployment process … and it contributed to their inability to respond to changing circumstances; you get the Kubernetes complexity and none of the Kubernetes advantages.

Re: A skeptic's first contact with Kubernetes

#103

Post by jauntywundrkind » Sun, Aug 04, 2024, 6:40 AM UTC

hbogert wrote: post #7

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…

There's a ton of discussion of alternative ideas as comments. Many look amazing.

But I keep wondering how many would be useful in giving us the modularity to make something like the shareable charts Helm has. That bitnami and others have these massive troves of charts is such a superpower for Kubernetes.

We all seem to agree it's not the string templating that's excellent, but figuring out how to package & make modular the templates/generators, and how to manage the outputs of the generators: that's "the rest of the owl" that tools such as Kustomzie and languages such as Dhall and starlark don't really buy us.

There is MetaController, which seems like some way to operationalize running generators and managing outputs. But I'm not sure what good examples there are of its use and specifically what examples are intended for distribution alike Helm Charts sometimes are.

Re: A skeptic's first contact with Kubernetes

#104

Post by hbogert » Mon, Aug 05, 2024, 6:22 AM UTC

FridgeSeal wrote: post #40

hbogert wrote: post #7

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody mak…

> I've found solace in CUE We rebuilt our Kustomization layers to be built via cue, so that we could theoretically work with less mistakes. Oh my god, never again. Incredibly confusing and horrible errors. Writing new modules was painful and unclear, trying to compose them together was even worse. I still get flashbacks to that “Invalid bytes” error.

Man I'd love some real war story sharing here! This might save me some time. B/c I have to admit i'm still in my honeymoon period with CUE

Re: A skeptic's first contact with Kubernetes

#105

Post by llarsson » Tue, Aug 06, 2024, 1:38 PM UTC

deivid wrote: post #36

hosh wrote: post #22

This is a good start. It misses an important thing about Kubernetes that is often missed: extensibility. Each and every thing within Kubernetes can be swapped out for something else, including the scheduler. There are Custom Resource Definitions that supports these extensions and operators. For example, there is no built-in autoscaler for nodes, but someone wrote one and you can add one in there. It uses a constraint…

Thanks for the detailed answer > extensibility This is something that irks me right now, but at my current knowledge level (~0) it didn't feel right to even mention: If Kubernetes pushes everything out (CSI, CNI, Cloud provider integration, LoadBalancer, ...), doesn't it become "just a control loop runner"? I'm sure there's value in that, but I can imagine that most people running clusters would make different choice…

That's right, and it's also why you see Kubernetes distributions popping up.

That way, someone has already done all the configuration of various plugins and components for you. For instance, the major clouds that all let you easily start with their Kubernetes services and that they integrate well with the logging and monitoring systems, IAM, server/node provisioning, etc.

Or ones that are not tied to any particular cloud provider, and perhaps focused on security (full disclosure: I work for Elastisys, who makes exactly that) or ease of use.

I'm sure we will see more and more such efforts in this space, exactly because cobbling together something yourself from scratch (basically just a control loop runner, as you said) is neither very appealing when you think about ongoing maintenance, nor very cost-effective for businesses to spend engineering time on.