Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we learned after a year on Kubernetes

about.gitlab.com

Post reply on HN
Print view View original

31–40 of 162 posts

Re: What we learned after a year on Kubernetes

#31

Post by magnawave »

cpaika wrote: post #19
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

Helm is amazing. If you are migrating legacy applications that rely on filesystem config, you can publish that config in a helm chart that applications require as sub charts. Gives you the ability to do centralized, versioned config management for legacy workloads going to the cloud Declarative paradigms get a bad rap but I really just believe its people who missing imperative/functional programmimg. Theres a reason…

The declarative functionality comes from K8s not helm. (Of course K8s has imperative tooling too but that’s not the point here).

The main reason folks dislike helm are 2 fold: 1) using string templating on a space formatted language is just messy and ugly for no good reason. If you don’t do anything terribly complicated it’s not too bad but quickly can be pretty fragile and unreadable. 2) golang(which I like generally) but the templating is messy and adds to the hard to read later problem

Personally find helm’s opportunity for code reuse to be restrictive and folks just end up cut and pasting a lot of declarations all over which is ... so terrible. But the first bullet above is pretty a fundamental design flaw.

Re: What we learned after a year on Kubernetes

#32

Post by DuskStar »

ckdarby wrote: post #24
t3rabytes wrote: post #2

> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying. (with alb-ingress-controller you can change target-type to IP to have traffic go direct…

What you're looking for is called Service Topology it is in Alpha. https://kubernetes.io/docs/concepts/services-networking/serv...

I genuinely cannot wait for that to make it to EKS.

Re: What we learned after a year on Kubernetes

#33

Post by aeontech »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Have you looked at Dhall or Tanka? They are both designed to solve exactly this problem (and I couldn’t agree with you more)

Re: What we learned after a year on Kubernetes

#34

Post by knocte »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

> .. with code

Use Pulumi, it does exactly this.

Re: What we learned after a year on Kubernetes

#35

Post by ckdarby »

DuskStar wrote: post #32
ckdarby wrote: post #24

Earlier quoted context omitted.

What you're looking for is called Service Topology it is in Alpha. https://kubernetes.io/docs/concepts/services-networking/serv...

I genuinely cannot wait for that to make it to EKS.

Yep, we were getting an insane amount of AZ chatter with Apache Pulsar because of all the components:

Request(us-west-1a) -> Pulsar Proxy(us-west-1b) -> Broker(us-west-1c) -> Bookie (us-west-1a)

Though for anyone using Pulsar on EKS who need multi-az for disaster recovery for the time being could deploy in a single AZ and then use the geo-replication feature in Pulsar to copy over to another AZ.

Re: What we learned after a year on Kubernetes

#36

Post by ecnahc515 »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

I generally agree. I'm using kustomize right now, but I'm also using helm charts via a kustomize plugin called chartInflator. It allows us to re-use existing upstream boilerplate while still using kustomize. Though I've not been entirely happy with kustomize either, and I'm thinking I might begin to investigate jsonnet for some things.

Re: What we learned after a year on Kubernetes

#37

Post by argc »

creztoe wrote: post #14
dpc_pw wrote: post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](…

As helm charts become used by more people and more complicated, exposing more of the underlying config, they just turn into a set of yaml files with as much or more complexity as the thing they are trying to replace. Configuration with intelligence that allows arbitrary overrides of any of the underlying configuration is important in order to meet all use cases. Without that, helm will only be useful for a strict subset of use cases and eventually you will outgrow the chart, or the chart will grow in complexity until its worthless.

Re: What we learned after a year on Kubernetes

#38

Post by hardwaresofton »

gravypod wrote: post #23
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommen…

I actually use and prefer make + kustomize -- unfortunately the version of kustomize that is bundled with kubectl now is a bit behind on features sometimes but still it does "just enough" and I manage the rest with make.

Redistribution is important (and harder with a simple setup like make + kustomize) but I think obscuring the resources you need to run an application (custom or otherwise) is actually a bad thing for everyone in the ecosystem. If you're going to run something in your cluster, intimate knowledge of the resources required, the ways it changes your system is important.

Re: What we learned after a year on Kubernetes

#39

Post by dpc_pw »

aeontech wrote: post #33
dpc_pw wrote: post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Have you looked at Dhall or Tanka? They are both designed to solve exactly this problem (and I couldn’t agree with you more)

Honestly, I am not a fan of "configuration languages". Why do we need another bunch of languages. Were the existing languages not good enough? There's already shitload of them.

The only place where I find it useful is when the config is provider by an untrusted party. But even then scheme, lua would probably fit the bill.

Not strong feelings about it though, maybe I'm wrong.

All I want is a library for my favorite language, where I can easily output bunch of `yaml` files to feed to k8s. I guess pulumi is kind of like that, but I never had a chance to try it.

Re: What we learned after a year on Kubernetes

#40

Post by technics256 »

escardin wrote: post #28
creztoe wrote: post #14

Earlier quoted context omitted.

Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](…

I also think helm is terrible. Helm's stated goal is to be something of a package manager for apps in k8s, but this is fundamentally unworkable as shown by... Helm. It's hard to describe just how unworkable this idea is. Let's start with an example, you want to install an app (let's say Jira) and a DB backend of your choice, postgres or mysql. The first step where this all falls down is, it may or may not support you…

Thanks for the detailed post. Really helps newcomers like myself.

One of the benefits as a new user of k8s is the ability to grab a helm chart to get me most of the way with something like ELK. I want to go the way of Kustomize but can't seem to find the same with it?

Post reply on HN