What we learned after a year on Kubernetes • Hacker News Board

Re: What we learned after a year on Kubernetes

#111

Post by pjmlp » Thu, Sep 17, 2020, 11:53 AM UTC

Keycap wrote: post #101

pjmlp wrote: post #93

Earlier quoted context omitted.

30+ years of experience seeing fads come and go in this fashion driven industry.

Do you have any argument for it or is it a generic statement independent of anything? Because there have been other 'fads' still being strong today. Kubernetes solves real problems which have been hard for a long time. Its the first thing you, as an infrastructure team, want to have to be able to provide your teams a manageable environment for yourself. mesos, nomad, docker swarm and co. Developers don't want a VM an…

> ________ solves real problems which have been hard for a long time.

Please replace with Ansible, Chef, Puppet, HP Vault, Solaris Zones, OpenMosix, Bewolf, Grid Computing, MTS, CORBA, JEE .....

An argument based on seeing hype come and go, sellig conferences, books, certifications, consulting, training, and naturally one selfs curriculum.

Speaking of which,

"Kubernetes Certified Application Developer (CKAD) with Tests"

https://www.udemy.com/course/certified-kubernetes-applicatio...

Re: What we learned after a year on Kubernetes

#112

Post by loudlambda » Thu, Sep 17, 2020, 11:56 AM UTC

TylerE wrote: post #67

pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

Every time I read an article on k8 it feels like one of those last 90s XML “now you have two problems” jokes...

It's more way more then two. If you pick kubernetes you now have N problems (helm, istio, kustomize, etcd, kubeform, ...)

Re: What we learned after a year on Kubernetes

#113

Post by nuker » Thu, Sep 17, 2020, 12:04 PM UTC

Keycap wrote: post #110

nuker wrote: post #81

Earlier quoted context omitted.

> I learned not to use it and wait until the k8s fad goes away. Yup, pretty happy with our move from k8s to ECS. Almost there.

You are basically using a very similiar tool/workflow just as a managed service from Amazon with a hard dependency. This 'k8s' fad is the open source version of what you are using. Its like you would say 'monitoring is a fad' we migrated away from monitoring and are now using amazon CloudWatch and are super happy with it.

> This 'k8s' fad is the open source version of what you are using

That would be AWS EKS. ECS is very different beast. Wins big in infra-as-a-code (CFN). And simpler, fully integrated, and you get full AWS support, not the case with EKS or hand rolled kube.

> .. monitoring is a fad ..

We did some pretty fancy things in monitoring. ELK and more. And yes, it uses some useful bits of CW.

Re: What we learned after a year on Kubernetes

#114

Post by loudlambda » Thu, Sep 17, 2020, 12:05 PM UTC

mindwok wrote: post #71

bsaul wrote: post #55

Earlier quoted context omitted.

i'm curious to hear about your alternative deployment strategy (no snark intended)

Not OP, and don't even have an alternative, but I find it disappointing that kubernetes basically seems to be the only answer for 'running containers in production'. Docker and its tooling is pretty development focused, and Kubernetes is probably too complex for 95% of use cases, which makes me think there's gotta be something in the middle ground that reaps the benefits of containers, wraps them with good tooling, r…

AWS Fargate has worked really well for me. And only required an afternoon to get working. I've spent weeks learning kubernetes and still don't know enough to not hit rough edges.

Re: What we learned after a year on Kubernetes

#115

Post by kkapelon » Thu, Sep 17, 2020, 12:10 PM UTC

pjmlp wrote: post #63

bsaul wrote: post #55

Earlier quoted context omitted.

i'm curious to hear about your alternative deployment strategy (no snark intended)

VMs as usual.

Have you ever worked at a company that has more than 300 VMs that run more than 10 different apps?

Kubernetes solves a specific set of problems. Just because you personally don't have these problems, doesn't mean that Kubernetes is a fad.

Re: What we learned after a year on Kubernetes

#116

Post by kkapelon » Thu, Sep 17, 2020, 12:16 PM UTC

dpc_pw wrote: post #4

ishcheklein wrote: post #3

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…

> What I'd like is to generate serialized yaml files ... you know ... with code

Check out Pulumi (I am not affiliated with them in any way)

Re: What we learned after a year on Kubernetes

#117

Post by ex_amazon_sde » Thu, Sep 17, 2020, 12:18 PM UTC

kkapelon wrote: post #115

pjmlp wrote: post #63

Earlier quoted context omitted.

VMs as usual.

Have you ever worked at a company that has more than 300 VMs that run more than 10 different apps? Kubernetes solves a specific set of problems. Just because you personally don't have these problems, doesn't mean that Kubernetes is a fad.

I did, it's called Amazon. It does not need k8s or containers at all.

Re: What we learned after a year on Kubernetes

#118

Post by kkapelon » Thu, Sep 17, 2020, 12:20 PM UTC

ex_amazon_sde wrote: post #117

kkapelon wrote: post #115

Earlier quoted context omitted.

Have you ever worked at a company that has more than 300 VMs that run more than 10 different apps? Kubernetes solves a specific set of problems. Just because you personally don't have these problems, doesn't mean that Kubernetes is a fad.

I did, it's called Amazon. It does not need k8s or containers at all.

Great. So you also believe that Kubernetes is a fad and doesn't solve any problems for anybody and we should all stick with VMs?

Re: What we learned after a year on Kubernetes

#119

Post by aequitas » Thu, Sep 17, 2020, 12:25 PM UTC

TheDong wrote: post #56

bsaul wrote: post #55

Earlier quoted context omitted.

i'm curious to hear about your alternative deployment strategy (no snark intended)

For many simple applications, an aws autoscaling group and application load balancer is as good as k8s and far simpler. If you're running microservices (to the point where you have so many small services that having an asg per service), than the above strategy does waste resources compared to k8s I guess... But you'd need a lot of microservices to justify the significant overhead k8s has.

But now you have to manage those VM's, the OS lifecycle, different distributions, application deployment, configuration and dependencies. And we're back at containers. Or you have to go towards lambda.

For K8s you need to manage the vm's/os as well but that part is mostly trivial as it's highly focussed on immutable/disposable infrastructure.

Re: What we learned after a year on Kubernetes

#120

Post by escardin » Thu, Sep 17, 2020, 12:25 PM UTC

technics256 wrote: post #40

escardin wrote: post #28

Earlier quoted context omitted.

I also think helm is terrible. Helm's stated goal is to be something of a package manager for apps in k8s, but this is fundamentally unworkable as shown by... Helm. It's hard to describe just how unworkable this idea is. Let's start with an example, you want to install an app (let's say Jira) and a DB backend of your choice, postgres or mysql. The first step where this all falls down is, it may or may not support you…

Thanks for the detailed post. Really helps newcomers like myself. One of the benefits as a new user of k8s is the ability to grab a helm chart to get me most of the way with something like ELK. I want to go the way of Kustomize but can't seem to find the same with it?

If you want an ELK stack, you should look into the operators provided by Elastic [1]. All you need to do is write a very small manifest or two for each thing you want operated. I feel like this is a better solution to 'I want an ELK cluster' than a helm chart because it solves more problems without leaking.

[1] https://www.elastic.co/blog/introducing-elastic-cloud-on-kub...