Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we learned after a year on Kubernetes

about.gitlab.com

Post reply on HN
Print view View original

71–80 of 162 posts

Re: What we learned after a year on Kubernetes

#71

Post by mindwok »

bsaul wrote: post #55
pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

i'm curious to hear about your alternative deployment strategy (no snark intended)

Not OP, and don't even have an alternative, but I find it disappointing that kubernetes basically seems to be the only answer for 'running containers in production'. Docker and its tooling is pretty development focused, and Kubernetes is probably too complex for 95% of use cases, which makes me think there's gotta be something in the middle ground that reaps the benefits of containers, wraps them with good tooling, reproducibility, reliability, and observability, and avoids the complexity of k8s.

Re: What we learned after a year on Kubernetes

#72

Post by Keycap »

pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

Why would you think it would go away?

Its literaly the best option currently for what it is doing.

It has an unprecedented support behind it as well.

Multiply Vendors support it through a certificate k8s managed service.

It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair.

It allows for simple IaaC.

Is it young? Yes. Do we need more people with more expierence? yes.

Is this a problem? No.

Re: What we learned after a year on Kubernetes

#73

Post by kdkeyser »

dpc_pw wrote: post #39
aeontech wrote: post #33

Earlier quoted context omitted.

Have you looked at Dhall or Tanka? They are both designed to solve exactly this problem (and I couldn’t agree with you more)

Honestly, I am not a fan of "configuration languages". Why do we need another bunch of languages. Were the existing languages not good enough? There's already shitload of them. The only place where I find it useful is when the config is provider by an untrusted party. But even then scheme, lua would probably fit the bill. Not strong feelings about it though, maybe I'm wrong. All I want is a library for my favorite la…

My experience with using general purpose languages for things like configuration or build systems has not been great. You always seem to end up with building a bunch of abstractions on top of that language, which culminates in another in-house configuration/build DSL, for which no public info is available.

It is also very hard to prevent people from using the escape hatch of the general purpose language to "fix a problem quickly", while introducing unpredictable side effects.

Scons (using Python) and Gradle (using Groovy / Kotlin) are 2 cases where I have seen this go very wrong, very quickly. I now strongly prefer Maven, even with its warts and XML verbosity.

On the other hand, configuration languages also often suck, the abomination that is HCL (e.g used in Terraform) comes to mind (especially in its early days). Turns out it is really hard to make a good language, and a bad configuration language is even worse than using a general purpose language.

I do like Dhall though: sufficiently powerful so you never have to repeat yourself, while also preventing you from doing fancy stuff you will later regret, all with a very readable syntax, and the type system helps to catch typos / mistakes early. I now use it to generate all config files for tools that consume json/yaml, and often for other text based config files as well (where you can use it as a powerful type-safe template engine)

Re: What we learned after a year on Kubernetes

#74

Post by skytreader »

Traubenfuchs wrote: post #68
pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

My disappointment with Kubernetes as the next/current big thing is that it practically didn't change a thing for companies as a whole. You still need dedicated devops people / a dedicated team. There is so much to get wrong regarding security, reliability and performance. Most application developers stay away from k8 configuration and could not quickly replace a devops engineer. They are the wizards holding the power…

My experience is the same with the caveat that it made life somewhat easier for the devops team (of course, I was in the devops team here). All the common devops tasks were much easier (scaling, health checks, etc). Unfortunately, the least common ones were harder and lacked documentation to boot at least at first, until we got our own documentation to refer to.

To add to that, I think there's a very specific way you must do things to get K8s "right" and reap benefits. Or maybe it's a question of scale. There were honestly times when I thought maybe we could've stuck with good old servers. When I left the team was trying to redo microservices---this time properly---by introducing event queues and pubsub and finally split that database.

Love that workplace but I'm honestly glad I didn't get to participate in that transition. Sounds like "artificial complexity" as you put it. Would love to hear if they pulled it off though or not, a project postmortem basically.

Re: What we learned after a year on Kubernetes

#75

Post by spicybright »

pjmlp wrote: post #63
bsaul wrote: post #55

Earlier quoted context omitted.

i'm curious to hear about your alternative deployment strategy (no snark intended)

VMs as usual.

While I agree, I'd rather deal with the k8 nonsense than maintaining infrastructure to make multiple VMs load balance, fail and restart cleanly, etc.

Re: What we learned after a year on Kubernetes

#76

Post by smt88 »

Keycap wrote: post #72
pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…

"out of the box" is a huge stretch. So is "simple".

> Is it young? Yes. Do we need more people with more expierence? yes.

Of course this is a problem! It's costly (in time, money, and security) to pay your team to ramp up on Kubernetes.

The question is, what is the actual benefit? 99% of companies don't need it at all.

Re: What we learned after a year on Kubernetes

#77

Post by pjmlp »

spicybright wrote: post #75
pjmlp wrote: post #63

Earlier quoted context omitted.

VMs as usual.

While I agree, I'd rather deal with the k8 nonsense than maintaining infrastructure to make multiple VMs load balance, fail and restart cleanly, etc.

In almost 30 years of development, I never worked on Web scale Google style projects, and some of them did handle TB of data in a cluster.

The golden rule is not to pretend to work at a FANNG when not having the same problems.

Re: What we learned after a year on Kubernetes

#78

Post by AlphaSite »

dpc_pw wrote: post #4
ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

Carvel/k14s has been very nice for the usecase I had. Use ytt to generate the yams, using a yaml aware variant of skylark and kapp to manage the actual atomic deployment, delete, etc.

It’s very simple and worked quite well for me.

Disclaimer: work at VMware but on a different team (it’s an OS tool built by some of our engineers, which we use internally)

Re: What we learned after a year on Kubernetes

#79

Post by qubit23 »

t3rabytes wrote: post #2

> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying. (with alb-ingress-controller you can change target-type to IP to have traffic go direct…

Regarding the alb-ingress-controller blackholing traffic with an IP target type ingress, check out support for pod readiness gates: https://kubernetes-sigs.github.io/aws-alb-ingress-controller.... Assuming you're using a rolling deployment, old pods won't be stopped until the new ones are added to the ALB target group and are in a healthy state. This has eliminated the small number of ELB 500s we'd see on deploys.

Re: What we learned after a year on Kubernetes

#80

Post by the_duke »

Nullabillity wrote: post #26
creztoe wrote: post #14

Earlier quoted context omitted.

Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](…

> Could you explain why helm is garbage? Not the OP, but.. 1. YAML string templating makes it very easy to get indentation and/or quotation wrong, and the error messages can easily end up pretty far from the actual errors. Structured data should be generated with structured templating. 2. "Values" aren't typechecked or cleaned. 3. Easy to end up in a state where a failed deploy leaves you with a mess to clean up by h…

> "Values" aren't typechecked or cleaned.

Helm 3 does offer a solution: a JSONSchema definition file for the values.

Which works ... in a very Helm-like fashion. Meaning: it's messy and awkward.

Post reply on HN