What we learned after a year on Kubernetes • Hacker News Board

Re: What we learned after a year on Kubernetes

#51

Post by eaceaser » Thu, Sep 17, 2020, 6:36 AM UTC

gravypod wrote: post #23

ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommen…

For internal k8s config at our org we built an config DSL using Starlark. The golang Starlark interpreter is super easy to use and extend. Starlark is familiar to every developer in our org because we are a Python shop. The tooling then spits out k8s YAML.

Essentially the config language implementation would be the same logic that a helm chart would do, but you’re writing the logic in Go versus a text templating engine. You can easily unit test parts and rely on the compiler to catch basic mistakes. Way better than templating YAML.

We also provide escape hatches so people can easily patch the resources before they get serialized to YAML. People can use that to customize our standard deployment config however they want.

So far this has worked very well and been extremely easy to maintain.

Re: What we learned after a year on Kubernetes

#52

Post by pnathan » Thu, Sep 17, 2020, 6:38 AM UTC

ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

Helm is a wannabe package manager, designed for people who really like writing yaml instead of executable code. K8s doesn't really have a fully built out "apps" resource, so that starts the whole process down a gravelly slope of pain... with initial care and if your team need to groom their k8s configs, you can go a long way with helm before the pain starts. A particular problem with helm is that it isn't really a fully fledged programming language, but a glue laid over the golang templating system.

I use helm as little as possible. Usually only enough to template out a single file with the different bits of k8s infra stuck together with `---`.

If I was doing some relatively serious work on custom services with minimal needs to conform to best^H^H^H^Hpopular practices, my service's configuration would be a neatly packaged Java program (its libraries for K8s direct API access are good).

There are different approaches to handling ops tasks; I favor the ones that are rooted in writing executable code over the ones with yaml.

Some bright chappie will probably write a Common Lisp library derivation library and k8s will be obvious and simple with that interface, but I'm not that bright - all I can see is the potential.

Re: What we learned after a year on Kubernetes

#53

Post by eaceaser » Thu, Sep 17, 2020, 6:45 AM UTC

SomaticPirate wrote: post #25

gravypod wrote: post #23

Earlier quoted context omitted.

Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommen…

Dhall continues to a be a glimmer of hope in this space. I agree that helm caused the entire space to distort. There is so little reusability in charts. Dhall seems to be the start of reversing this, but I hesitate to mention it since the ergonomics are still rough. If YAML is the assembly language of k8s, Dhall looks like what you would write the compiler in.

IMO because Dhall is unique syntax to the vast majority of developers who have never done anything with Haskell or Elm or whatever, it’s a non starter for large case use. We use Starlark, where the syntax is familiar to anyone familiar with python.

I like your analogy though. For us, the assembly language is using the k8s API directly in Golang. The “compiler” is the golang Starlark interpreter extended with our own config API, like you would implement in Dhall. It’s just in this case, you can implement it in Golang, which has much much better tooling than Dhall does. A typed compiler, debugger, IDE, unit tests.... so much easier to develop and maintain.

Re: What we learned after a year on Kubernetes

#54

Post by pjmlp » Thu, Sep 17, 2020, 6:45 AM UTC

I learned not to use it and wait until the k8s fad goes away.

Re: What we learned after a year on Kubernetes

#55

Post by bsaul » Thu, Sep 17, 2020, 6:51 AM UTC

pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

i'm curious to hear about your alternative deployment strategy (no snark intended)

Re: What we learned after a year on Kubernetes

#56

Post by TheDong » Thu, Sep 17, 2020, 6:55 AM UTC

bsaul wrote: post #55

pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

i'm curious to hear about your alternative deployment strategy (no snark intended)

For many simple applications, an aws autoscaling group and application load balancer is as good as k8s and far simpler.

If you're running microservices (to the point where you have so many small services that having an asg per service), than the above strategy does waste resources compared to k8s I guess... But you'd need a lot of microservices to justify the significant overhead k8s has.

Re: What we learned after a year on Kubernetes

#57

Post by jarv » Thu, Sep 17, 2020, 7:03 AM UTC

ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

Blog author here - we are using Helm for our application because of how much we have already invested in our public Helm chart.

We are not sold on using Helm however for K8s in general and are currently evaluating Tanka for some of our monitoring infrastructure. One of our engineers wrote up some notes on that here if you are curious https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issu...

Re: What we learned after a year on Kubernetes

#58

Post by FridgeSeal » Thu, Sep 17, 2020, 7:09 AM UTC

creztoe wrote: post #14

dpc_pw wrote: post #4

Earlier quoted context omitted.

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](…

We've found Kustomize, or just straight up writing the deployments ourselves the best approach.

The actual spec for a Deployment/Daemon Set/Stateful Set/CRD is usually super straightforward once you get the Kubernetes terminology, and most of the issues I've had with Helm have boiled down to "oh they haven't parametised the one config I need to change"

Re: What we learned after a year on Kubernetes

#59

Post by stepbeek » Thu, Sep 17, 2020, 7:10 AM UTC

Was anyone else surprised by the timeline of this post? I was under the impression that Gitlab had been fully on K8S for years now, given the hard push of their K8S offering 18 months ago.

Re: What we learned after a year on Kubernetes

#60

Post by jarv » Thu, Sep 17, 2020, 7:19 AM UTC

stepbeek wrote: post #59

Was anyone else surprised by the timeline of this post? I was under the impression that Gitlab had been fully on K8S for years now, given the hard push of their K8S offering 18 months ago.

It has been possible to run GitLab the application on K8s for some time now for the majority of self-managed deployments. There were some limitations however, for running GitLab on K8s at the very large scale of GitLab.com that we have been working through in the last year.

(disclaimer: blog post author)