Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we learned after a year on Kubernetes

about.gitlab.com

Post reply on HN
Print view View original

11–20 of 162 posts

Re: What we learned after a year on Kubernetes

#11

Post by esprehn »

justicezyx wrote: post #7
dpc_pw wrote: post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Insider joke: K8s was born to solve the same problem Borg solved or simply design in such a way to not have that problem. One of them is the config files hell. Borgcfg files ranks #3 in terms of line count across all of Google. (Borgcfg language is similar to Jsonnet, helm) In terms of the amount of power wielded buy a single CLI, `borgcfg`, this little program that has less than 50k lines of nontesting code, stands…

Relevant thread (with you) from the nearby past: https://news.ycombinator.com/item?id=20849454

The yaml monster in k8s is quite unpleasant. It's always been surprising to me how we somehow ended up with executable BUILD files but totally static deploy files.

Re: What we learned after a year on Kubernetes

#12

Post by richm31415 »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Or eschew config files altogether and write operators in golang . . . https://sdk.operatorframework.io/

Re: What we learned after a year on Kubernetes

#13

Post by tootie »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

If you're doing more than storing config values in YAML, you're playing with fire. I've really grown to dislike the trend in "developing by descriptor". It's great when it works and impossible to debug when anything goes wrong. And also, not easily testable.

Re: What we learned after a year on Kubernetes

#14

Post by creztoe »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements.

Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](https://github.com/kubernetes-client/python) without much trouble. Personally I prefer editing them as JSON instead of python objects, but it isn't too bad.

Re: What we learned after a year on Kubernetes

#15

Post by dilyevsky »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Sounds like you may enjoy https://github.com/cruise-automation/isopod (some assembly may be required)

Re: What we learned after a year on Kubernetes

#16

Post by gscho »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

You can't just say helm is garbage without saying why.

Re: What we learned after a year on Kubernetes

#17

Post by gscho »

AaronFriel wrote: post #9
dpc_pw wrote: post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Are you familiar with Pulumi? It does away with the notion that you need serialized yaml files, although in its current incarnation it can produce them, and instead runs programs which generates collections of named (and hierarchically defined) resources which can then be "applied" as a batch. It uses lazy values to generate a directed graph to determine the order for creating and tearing down resources as well as sa…

...and you need to use their SaaS to make it work, no?

Re: What we learned after a year on Kubernetes

#18

Post by bfrydl »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

I don't know if I think Helm is garbage but I feel like it's almost always more trouble than its worth. Most charts can be boiled down to just a couple resources and instead of sifting through its weirdo templates you might as well just look at the actual resource configs. It's almost always just a single Deployment or similar + RBAC.

I've actually been using Terraform to manage Kubernetes resources at my latest job, along with literally everything else. I don't even use alb-ingress-controller or external-dns or any of that. I just have Terraform make target groups from the service resources. It breaks a lot less.

Re: What we learned after a year on Kubernetes

#19

Post by cpaika »

ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

Helm is amazing. If you are migrating legacy applications that rely on filesystem config, you can publish that config in a helm chart that applications require as sub charts. Gives you the ability to do centralized, versioned config management for legacy workloads going to the cloud

Declarative paradigms get a bad rap but I really just believe its people who missing imperative/functional programmimg. Theres a reason the infra space has been using declarative syntax for years now

Re: What we learned after a year on Kubernetes

#20

Post by OJFord »

bfrydl wrote: post #18
dpc_pw wrote: post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

I don't know if I think Helm is garbage but I feel like it's almost always more trouble than its worth. Most charts can be boiled down to just a couple resources and instead of sifting through its weirdo templates you might as well just look at the actual resource configs. It's almost always just a single Deployment or similar + RBAC. I've actually been using Terraform to manage Kubernetes resources at my latest job,…

But Helm barely adds any boilerplate over that 'single deployment or similar + RBAC'? Just a Chart.yaml (minimal of which is very small) and adding the option to parameterise or templatise the resources.

I don't know why you wouldn't, we don't say 'I don't know if .deb/.rpm/PKGINFO/etc. is worth it, most packages are just a binary'.

For Helm to add LoC you'd have to all but not use it, and even then it'd be about 3LoC. Setup is no harder, with N kubectl applies becoming 1 helm install.

I also started using terraform (directly instead of just triggering helm) - are you using kubernetes tf resources directly though, or the helm provider?

Post reply on HN