Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we learned after a year on Kubernetes

about.gitlab.com

Post reply on HN
Print view View original

101–110 of 162 posts

Re: What we learned after a year on Kubernetes

#101

Post by Keycap »

pjmlp wrote: post #93
Keycap wrote: post #72

Earlier quoted context omitted.

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…

30+ years of experience seeing fads come and go in this fashion driven industry.

Do you have any argument for it or is it a generic statement independent of anything?

Because there have been other 'fads' still being strong today.

Kubernetes solves real problems which have been hard for a long time. Its the first thing you, as an infrastructure team, want to have to be able to provide your teams a manageable environment for yourself.

mesos, nomad, docker swarm and co.

Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them.

Cloud native is here to stay; It will affect and already affects tools, applications and architecture.

Re: What we learned after a year on Kubernetes

#102

Post by mkingston »

Nullabillity wrote: post #84
alephu5 wrote: post #48

Earlier quoted context omitted.

It's not going to solve all your problems but dhall can fix your first few gripes. I've been using it for several months and it's an excellent way to write configuration imo.

Yeah, I have used Nix to generate them in the past, which worked pretty great too. But Helm does, admittedly, solve a real problem: garbage collecting old resources when they're deleted from the repo. I just wish we could have something much simpler that only did that...

`kubectl apply --prune` should nominally do this. Irritatingly (I acknowledge I'm almost as responsible as anyone else for doing something about this), it's had this disclaimer on it for quite some time now:

> Alpha Disclaimer: the --prune functionality is not yet complete. Do not use unless you are aware of what the current state is. See ⟨https://issues.k8s.io/34274⟩.

I haven't used it in anger, so I can't add any disclaimer or otherwise of my own.

kpt is the recent Google-ordained (AFAICT) solution to this problem, but is ot yet at v1.

You could also resolve this yourself by either:

* versioning with labels and deleting all resources with labels indicating older versions

* using helm _only_ in the last mile, for pruning

Re: What we learned after a year on Kubernetes

#103

Post by Keycap »

smt88 wrote: post #76
Keycap wrote: post #72

Earlier quoted context omitted.

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…

"out of the box" is a huge stretch. So is "simple". > Is it young? Yes. Do we need more people with more expierence? yes. Of course this is a problem! It's costly (in time, money, and security) to pay your team to ramp up on Kubernetes. The question is, what is the actual benefit? 99% of companies don't need it at all.

The features i mentioned, are solved 'out of the box'. I'm not seeing a huge stretch here at all.

Its costly and risky to run 100 VMs, maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed.

It is cost ineffective to have security audits on 100 VMs, maintaining access to them, auditing whats happening on them.

It is a ton easier to allow someone only access to one namespace, limited ingress domains (which get provisioned automatically) and allow them to only run non root containersl

The question is not what the actual benefit is (its clear and i mentioned it in my paragraphs before), the question is will kubernetes ever become so lightweight, stable, easy to use so that its feasable for normal people to run it on a 3-5 node cluster in small companies.

And please lets ignore all those small companies where people log into their VMs by hand and maintaining them by using snapshots and cloning existing VMs. Those small companies might and should just migrate to cloud managed services completly.

Re: What we learned after a year on Kubernetes

#104

Post by secondcoming »

thu2111 wrote: post #89
spicybright wrote: post #75

Earlier quoted context omitted.

While I agree, I'd rather deal with the k8 nonsense than maintaining infrastructure to make multiple VMs load balance, fail and restart cleanly, etc.

systemd will happily health check a service, restart it when it fails and so on. The basics don't need much. Auto-scaling is something I'm suspicious of. I saw many experiments with that at Google when I worked there, under the name of elasticity. They found the same thing GitLab found here: it's really hard and the expected savings often don't materialise. Even in 2012 their elasticity projects were far more advance…

I found AWS's autoscaling offering be better than GCP's, in that you could use custom CloudWatch metrics as a basis for scaling logic rather than just the standard metrics (CPU usage, latency, etc)

GCP does offer custom metrics, but I couldn't figure out how to tie them to the LB scaling logic.

Re: What we learned after a year on Kubernetes

#105

Post by secondcoming »

Keycap wrote: post #101
pjmlp wrote: post #93

Earlier quoted context omitted.

30+ years of experience seeing fads come and go in this fashion driven industry.

Do you have any argument for it or is it a generic statement independent of anything? Because there have been other 'fads' still being strong today. Kubernetes solves real problems which have been hard for a long time. Its the first thing you, as an infrastructure team, want to have to be able to provide your teams a manageable environment for yourself. mesos, nomad, docker swarm and co. Developers don't want a VM an…

> Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them.

I want a VM because I want my dev environment to exactly replicate the production environment, down to the kernel. It also means that if I leave my employer I can just delete my VM on my personal machine.

We've also not had any issues with maintaining production VMs, our release pipeline packages up our code into a debian package and them creates a VM image from that. This is then deployed across all regions. If you need to roll back just deploy the previous VM image version. GCP handles killing old instances and starting new instances for you.

Re: What we learned after a year on Kubernetes

#106

Post by Keycap »

thu2111 wrote: post #88
Keycap wrote: post #72

Earlier quoted context omitted.

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…

I dunno, because of writeups like GitLabs? We're lucky GitLab is so transparent. It lets us see that their claims don't appear to always match the actual reality. Here's an example. Blog post says: "After transitioning each service, we enjoyed many benefits of using Kubernetes in production, including much faster and safer deploys of the application, scaling, and more efficient resource allocation" Wrong. Actual anal…

Its really weird that your argument is based on just the cost.

I mean you mentioned yourself you worked at google right?

Perhaps you just haven't actually experienced the issues kubernetes is solving?

How often have you seen that certificates expired? I have seen that. Plenty of times. Its not an issue creating that lets encrypt cronjob, its still something you need to do right.

Security Updates? Have you seen how many companies run with old non updated VMs?

Disk full due to logs? Yes seeing this regularly.

Memory leak on a service and someone needs to restart it manually until someone else fixes the issue? Yes!

Requesting a VM, hardware, infrastructure, getting it and the whole lifecycle management of it in the backend? Its real.

Ansible Scripts, puppet or just bash scripts and a word document to tell you how this magic machine was set up? Yepp.

Kubernetes solves all those problems.

Your static website on borg, if it sill runs, has probably still a valid certificate, is running on a secure infrastructure, is equally configured on every instance and not weird on 1 of 6 servers and just runs.

A smart person taking responsibility for all of this, costs you much more then just a few hundred bucks a month. And you need that person. With Kubernetes, this person now can manage and operate much more servers under his/her fingertips better easier and more secure then if it would have been vms.

And in my personal experience: That shit runs more stable because that shit can restart and being recreated and it just solves a handfull of shitty memory or disk full issues.

Re: What we learned after a year on Kubernetes

#107

Post by cpursley »

regularfry wrote: post #64
zelphirkalt wrote: post #49

I do have only very minimal xp in k8s, as I have tried to use it only once (and never again). To me it seem like mOst of the comments are about the unusable and unmaintainable yaml configuration files. This seems like exactly the issue I had a few years ago, when I was cursing, because I could not find a spec or docs for the yaml stuff in all the pages and pages of seemingly important documentation and thus could not…

There's definitely a need for an opinionated layer over the top, with much simpler configuration. Probably more than one, with different use cases. Nobody should have to wrangle that much yaml.

https://convox.com

It's basically heroku on top of k8s and you bring your own cloud.

Re: What we learned after a year on Kubernetes

#108

Post by Keycap »

secondcoming wrote: post #105
Keycap wrote: post #101

Earlier quoted context omitted.

Do you have any argument for it or is it a generic statement independent of anything? Because there have been other 'fads' still being strong today. Kubernetes solves real problems which have been hard for a long time. Its the first thing you, as an infrastructure team, want to have to be able to provide your teams a manageable environment for yourself. mesos, nomad, docker swarm and co. Developers don't want a VM an…

> Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them. I want a VM because I want my dev environment to exactly replicate the production environment, down to the kernel. It also means that if I leave my employer I can just delete my VM on my personal machine. We've also not had any issues with maintaining production VMs, our release pipeline packages up our co…

I did not talk about your specific use case.

Feel free to be happy with your setup.

Also i'm not able to determine if your setup would still be much better (faster, easier to maintain etc.) if you would set completly on containers instead of a VM as i'm not aware of your workload at all.

Re: What we learned after a year on Kubernetes

#109

Post by pjmlp »

Traubenfuchs wrote: post #94
piokoch wrote: post #69

Earlier quoted context omitted.

Frankly K8 reminds me more J2EE EJB 2.X. Horrific complexity that provides not that much after all.

Kubernetes clearly, obviously and rather openly IS the application server of the current decade. I am saying this as a dev who has seen up close: OpenShift, Kubernetes, Tomcat and enterprise WebSphere. Now you can have all this complexity even in other languages than Java.

Hence why it is a fad, in about two years we will be getting the Spring/EJB 3.0 of Kubernetes, when enough people just get fed up with its complexity.

Re: What we learned after a year on Kubernetes

#110

Post by Keycap »

nuker wrote: post #81
pjmlp wrote: post #54

I learned not to use it and wait until the k8s fad goes away.

> I learned not to use it and wait until the k8s fad goes away. Yup, pretty happy with our move from k8s to ECS. Almost there.

You are basically using a very similiar tool/workflow just as a managed service from Amazon with a hard dependency.

This 'k8s' fad is the open source version of what you are using.

Its like you would say 'monitoring is a fad' we migrated away from monitoring and are now using amazon CloudWatch and are super happy with it.

Post reply on HN