Live data from Hacker News

What we learned after a year on Kubernetes

about.gitlab.com

91–100 of 162 posts

Re: What we learned after a year on Kubernetes

#91
post #69
post #67

Earlier quoted context omitted.

Every time I read an article on k8 it feels like one of those last 90s XML “now you have two problems” jokes...

Frankly K8 reminds me more J2EE EJB 2.X. Horrific complexity that provides not that much after all.

Almost everything that gets advertised about it I have mostly seen it first on application servers.

Re: What we learned after a year on Kubernetes

#92
post #54

I learned not to use it and wait until the k8s fad goes away.

My disappointment with Kubernetes as the next/current big thing is that it practically didn't change a thing for companies as a whole. You still need dedicated devops people / a dedicated team. There is so much to get wrong regarding security, reliability and performance. Most application developers stay away from k8 configuration and could not quickly replace a devops engineer. They are the wizards holding the power…

I agree. Through being complex, it stimulates management to turn devops into a separate pillar in the organisation, creating "devops" akin to "Agile".

The practical results for me were:

* Having to write a CLI app to reduce the time wasted and spit out simple config templates for our actual (very much not Google-scale) use case

* Whenever something didn't work, throw it over the wall and hope the DevOps gods deemed us worthy

I like tools that have sensible defaults and try to stay out of your way. K8s fails on both counts in my experience.

Re: What we learned after a year on Kubernetes

#93
post #72
post #54

I learned not to use it and wait until the k8s fad goes away.

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…

30+ years of experience seeing fads come and go in this fashion driven industry.

Re: What we learned after a year on Kubernetes

#94
post #69
post #67

Earlier quoted context omitted.

Every time I read an article on k8 it feels like one of those last 90s XML “now you have two problems” jokes...

Frankly K8 reminds me more J2EE EJB 2.X. Horrific complexity that provides not that much after all.

Kubernetes clearly, obviously and rather openly IS the application server of the current decade. I am saying this as a dev who has seen up close: OpenShift, Kubernetes, Tomcat and enterprise WebSphere.

Now you can have all this complexity even in other languages than Java.

Re: What we learned after a year on Kubernetes

#95
My question. Why not convert the rails code/APIs to blob/FaaS and skip all the DevOps complexity?

Are containers providing enough long-term cost-effectiveness to a complete cloud-native application architecture?

Or is it just mapping VMs to something “like” VMs so your topology remains mostly the same?

Re: What we learned after a year on Kubernetes

#97
post #88
post #72

Earlier quoted context omitted.

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…

I dunno, because of writeups like GitLabs? We're lucky GitLab is so transparent. It lets us see that their claims don't appear to always match the actual reality. Here's an example. Blog post says: "After transitioning each service, we enjoyed many benefits of using Kubernetes in production, including much faster and safer deploys of the application, scaling, and more efficient resource allocation" Wrong. Actual anal…

> Wrong. Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes.

This is only one of the shards of one service that we migrated, and we migrated many more.

I did a high level writeup of this and many other things we've observed in https://about.gitlab.com/handbook/engineering/infrastructure...

if you are curious. I am happy to have a conversation with anyone about our experiences in more detail, not to convince anyone that k8s is the "one-true-way" (because I am still not convinced myself), but of the benefits this type of change can bring.

> They tried to use auto-scaling but failed completely and ended up disabling it

That issue is just one of many though. We disabled it at the time, but we have it enabled for a number of other services.

Regardless of how I personally feel about K8s, I have to say that the migration we are doing for GitLab.com is generating set of benefits that goes far beyond just moving to a new platform.

One of the largest benefits I've seen so far is that it was a great forcing function to resolve some long running architectural challenges, and is making us think more about how the application can run at a very large scale, without being at the very large scale. Things that we could get away before, like the issue you referenced there, we can't anymore.

Disclaimer: I am one of the people involved in this migration.

Re: What we learned after a year on Kubernetes

#98
post #42

It reads like a press release, no gotchas, no complaints... https://k8s.af/

There are some complains that we can share, but honestly they are not really that fun to read. We are still early in our transition and one major complaint I have is about helm and lack of flexibility of it and operator being very flexible but a huge time sink.

I think a lot of complaints we would usually have are reduced due to the fact that we use managed K8s with GKE.

All other complaints I can think of are related to our application and some architecture decisions we made early on, but that won't be interesting to anyone but people at GitLab.

Re: What we learned after a year on Kubernetes

#99
> We found that with an application that increases its memory utilization over time, low requests (which reserves memory for each pod) and a generous hard limit on utilization was a recipe for node saturation and a high rate of evictions. To adjust for this we eventually decided to use higher requests and lower limit which took pressure off of the nodes and allowed pods to be recycled without putting too much pressure on the node.

Kubernetes works at its most efficient when you scale by adding more Pods, rather than scaling within a Pod. Requests and limits should be equal to each other, and a HorizontalPodAutoscaler (or KEDA Scales object) should be used to add more Pods as utilization increases. Using Cluster Autoscaler then makes sure that your cluster has sufficient Nodes to schedule the Pods. Then you just need to set up alerting to tell you if the autoscaling reaches minimums (indicating that you should consider lowering the minimum, to improve efficiency) or maximums (indicating that you should consider increasing the maximums), and you can leave things well enough alone, unless the speed at which scaling occurs is insufficiently slow for your use case.

Re: What we learned after a year on Kubernetes

#100

I do have only very minimal xp in k8s, as I have tried to use it only once (and never again). To me it seem like mOst of the comments are about the unusable and unmaintainable yaml configuration files. This seems like exactly the issue I had a few years ago, when I was cursing, because I could not find a spec or docs for the yaml stuff in all the pages and pages of seemingly important documentation and thus could not…

This. I'm currently trying to figure this mess out and all I want is a reference for the different objects and what parameters they take, or what the hell an "apiVersion" is and which ones are available. Instead their "documentation" is a bunch of unhelpful guides that never show examples of what I need to know. How this has reached widespread adoption is beyond me.
Post reply on HN