Live data from Hacker News

What we learned after a year on Kubernetes

about.gitlab.com

21–30 of 162 posts

Re: What we learned after a year on Kubernetes

#21
post #17

Earlier quoted context omitted.

Are you familiar with Pulumi? It does away with the notion that you need serialized yaml files, although in its current incarnation it can produce them, and instead runs programs which generates collections of named (and hierarchically defined) resources which can then be "applied" as a batch. It uses lazy values to generate a directed graph to determine the order for creating and tearing down resources as well as sa…

...and you need to use their SaaS to make it work, no?

I don't think so, but you need to manage/sync the infrastructure state yourself.

Re: What we learned after a year on Kubernetes

#22
post #4

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

> Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me.

Someone here talked about the joys of using the API with protobufs. Something I want to try out because it sounds clean.

Re: What we learned after a year on Kubernetes

#23

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug.

Kustomize, too, doesn't get at the root of all the needs I've found myself having.

A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommend kubecfg [0] as it's a tool that "just works" and is built at the perfect level of abstraction: "just give me something and I'll find all the kubernetes objects inside of it". It's remarkably simple and forgiving and it makes it possible to describe even the most complex application configs + kube configs in a single language. You can also take it a step further and describe your CI stages (pre-merge, release, builds, etc), kube yaml, and infrastructure provisioning (terraform) all in one language.

> What are the benefits of using helm?

The main benefit is redistribution. Since everyone uses helm it's easy to give someone a chart because they know what to do with it. That's pretty much the only upside I've found.

[0] - https://github.com/bitnami/kubecfg

Re: What we learned after a year on Kubernetes

#24

> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying. (with alb-ingress-controller you can change target-type to IP to have traffic go direct…

What you're looking for is called Service Topology it is in Alpha.

https://kubernetes.io/docs/concepts/services-networking/serv...

Re: What we learned after a year on Kubernetes

#25

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommen…

Dhall continues to a be a glimmer of hope in this space. I agree that helm caused the entire space to distort. There is so little reusability in charts. Dhall seems to be the start of reversing this, but I hesitate to mention it since the ergonomics are still rough.

If YAML is the assembly language of k8s, Dhall looks like what you would write the compiler in.

Re: What we learned after a year on Kubernetes

#26
post #14
post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](…

> Could you explain why helm is garbage?

Not the OP, but..

1. YAML string templating makes it very easy to get indentation and/or quotation wrong, and the error messages can easily end up pretty far from the actual errors. Structured data should be generated with structured templating.

2. "Values" aren't typechecked or cleaned.

3. Easy to end up in a state where a failed deploy leaves you with a mess to clean up by hand.

4. No good way to preview what a deploy will change.

5. Weird interactions when resources are edited manually (especially back in Helm 2, but still a thing).

6. No good way to migrate objects into a Helm chart without deleting and recreating them.

7. Tons of repetitive boilerplate in each chart to customize basic settings (like replica counts).

It's a typical Go solution, in all the wrong ways.

Re: What we learned after a year on Kubernetes

#27
post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Or eschew config files altogether and write operators in golang . . . https://sdk.operatorframework.io/

Still have to write a CRD if you want to do anything useful (arguably still requires a config).

Re: What we learned after a year on Kubernetes

#28
post #14
post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](…

I also think helm is terrible.

Helm's stated goal is to be something of a package manager for apps in k8s, but this is fundamentally unworkable as shown by... Helm. It's hard to describe just how unworkable this idea is.

Let's start with an example, you want to install an app (let's say Jira) and a DB backend of your choice, postgres or mysql.

The first step where this all falls down is, it may or may not support your preferred DB. Sure, Jira does, but does the chart?

Assuming it does support your preferred backend, maybe it depends on the chart for the db you picked. If it does, it's going to install it for you, hopefully with best practices, almost certainly not according to your corporate security policy. This is also a problem if say, you have a db you want to use already, prefer to use an operator for managing your db, or use a db outside k8s.

You got lucky, and it supports your DB just the way you want it. Next question, do you want an HA Jira? Often, this part is done so differently that HA Jira and Single host Jira are straight up different charts.

Do you want your DB to be HA? Unfortunately, the chart Jira chart author picked to depend on is the non-HA one. Guess you're out of luck.

Maybe you want to add redis caching? Nginx frontend/ingress? Want to terminate TLS at the app host and not ingress? How do you integrate it with your cert management system?

We haven't even looked at the config, where you have to do everything in a variables.yaml file which is never documented as well as the actual thing it's configuring on your behalf, and is not always done in a sensible manner.

Hopefully it's clear from this that as a user, helm isn't going to work for you, because just as there's no such thing as the average person, there's no such thing as an average deployment. Even a basic one is filled with one off variations for every user that a public chart needs to support.

As a developer, helm is unworkable because you're templating yaml inside yaml. This isn't too bad if you're just tweaking a few things on an otherwise plain chart, but a public chart, that naively hopes to support all the possible configurations? Your otherwise simple chart is now 5-10x longer from all the templating options. Have fun supporting it and adding the new features you'll inevitably need to add and support.

As a counterpoint to all this, kustomize gets a lot right. I don't mean that kustomize is perfect, or even good, but I've found like k8s itself, it understands that the problem space is complex and to try and hide that complexity leads to a lesser product that is more complex because of leaky abstractions.

Kustomize acts as a glue layer for your manifests, so instead of some giant morass of charts and dependencies none of which work for you, you're expected to find a suitable chart for each piece yourself and compose them with Kustomize.

Going through the same example again as a user:

Your vendor has provided a couple basic manifests for you to consume, maybe even only one, because they're expecting you to supply your own DB. Since they only need to supply the Jira part, instead of having an HA chart and a Single node manifest, They just give you one manifest with a stateful set. Or maybe they give you two. One as a deployment and one as a Stateful set. The stateful set might also have an upgrade policy configured for you.

Since the vendor punted on the DB, you can do whatever you like here. You'll have to supply a config map with your db config to the Jira deployment, but that's okay, it's easy to override the default one in the manifest with kustomize. You are now free to use a cloud managed DB, an operator managed one, or just pull the stock manifest for your preferred DB.

Want to terminate TLS in your app again? Easy enough, Cert Manager will provide the certs for you, and supply them as secrets, ready to consume in your app, you just need to tell it where to look.

So now you have all the parts of your Jira deployment configured just how you like, but they're all separate. Maybe you just edited the stock manifests to get your changes in the way you like. Dang, you've created a problem when the vendor updates the manifest, as now you need to merge in changes every time you update it. That seems like a huge hassle. What if you could just keep your changes in a patch and apply that to a stock manifest? Then it's easy to diff upgrades of the stock manifest and see what changed, and it's easy to see what you care about in your patch.

All of this seems like it's getting kind of unwieldy, maybe we can make it easier. We'll have a single kustomization.yaml, and it'll be really structured. In it, you can list all the paths to folders of manifests, or individual files, or git repos/branches/subpaths. We'll also specify a set of patches that look like partial manifests to apply to these base manifests to make it clear what is the base and what goes on top. Then finally, for common things like image versions and namespaces, we'll expose them directly so you don't need to patch everything. We can do that because we're using standard manifests that can be parsed and modified.

That is kustomize, and as awkward as it is, it's just trying to make it clear what you're applying, and what your one customization from stock is in a maintainable way. It does a better job of 'package management' by not managing packages. This is pretty similar to how Linux package managers work. If you install wordpress on your server, it's going to install php, but it might not install apache or nginx, as everyone wants something different. It's definitely not going to install a DB for you. You as the admin have to decide what you want and tell each part about the other.

Re: What we learned after a year on Kubernetes

#29

Earlier quoted context omitted.

Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommen…

Dhall continues to a be a glimmer of hope in this space. I agree that helm caused the entire space to distort. There is so little reusability in charts. Dhall seems to be the start of reversing this, but I hesitate to mention it since the ergonomics are still rough. If YAML is the assembly language of k8s, Dhall looks like what you would write the compiler in.

Yea, anything that allows you to think in a higher-level than raw kube objects will save you a lot of time and heartache. For instance, if your application needs a redis cluster in helm you have to mess with subcharts/other headaches. In jsonnet it becomes `redis = import "redis"; redis.Cluster("my-app-name") { nodes: 10 }` or something similar.

DRY configs make me very happy.

Re: What we learned after a year on Kubernetes

#30

Earlier quoted context omitted.

Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommen…

Dhall continues to a be a glimmer of hope in this space. I agree that helm caused the entire space to distort. There is so little reusability in charts. Dhall seems to be the start of reversing this, but I hesitate to mention it since the ergonomics are still rough. If YAML is the assembly language of k8s, Dhall looks like what you would write the compiler in.

Given that the Dhall type system supports things that the other configuration languages do not, isn't it very hard to migrate to for a project like k8s?

I don't see how they could support something in parallel to Dhall, and still enable all the features Dhall provides.

Post reply on HN