I learned not to use it and wait until the k8s fad goes away.
> I learned not to use it and wait until the k8s fad goes away.
Yup, pretty happy with our move from k8s to ECS. Almost there.
What we learned after a year on Kubernetes
81–90 of 162 posts
Re: What we learned after a year on Kubernetes
#82Re: What we learned after a year on Kubernetes
#83Earlier quoted context omitted.
While I agree, I'd rather deal with the k8 nonsense than maintaining infrastructure to make multiple VMs load balance, fail and restart cleanly, etc.
In almost 30 years of development, I never worked on Web scale Google style projects, and some of them did handle TB of data in a cluster. The golden rule is not to pretend to work at a FANNG when not having the same problems.
I agree, but man so many places are going in the opposite direction… I got lucky with my current gig that I got to implement load balancing/autoscaling with Fabric, ~1000 lines of python in a daemon using boto3, nginx and disk images…
Re: What we learned after a year on Kubernetes
#84Earlier quoted context omitted.
> Could you explain why helm is garbage? Not the OP, but.. 1. YAML string templating makes it very easy to get indentation and/or quotation wrong, and the error messages can easily end up pretty far from the actual errors. Structured data should be generated with structured templating. 2. "Values" aren't typechecked or cleaned. 3. Easy to end up in a state where a failed deploy leaves you with a mess to clean up by h…
It's not going to solve all your problems but dhall can fix your first few gripes. I've been using it for several months and it's an excellent way to write configuration imo.
Yeah, I have used Nix to generate them in the past, which worked pretty great too. But Helm does, admittedly, solve a real problem: garbage collecting old resources when they're deleted from the repo. I just wish we could have something much simpler that only did that...
Re: What we learned after a year on Kubernetes
#85Earlier quoted context omitted.
`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ...…
I don't know if I think Helm is garbage but I feel like it's almost always more trouble than its worth. Most charts can be boiled down to just a couple resources and instead of sifting through its weirdo templates you might as well just look at the actual resource configs. It's almost always just a single Deployment or similar + RBAC. I've actually been using Terraform to manage Kubernetes resources at my latest job,…
Most helm charts I have seen and used have a deployment yaml or statefulset or daemonset plus a service yaml, optional ingress yaml, sidecar containers for metrics, and several others I'm forgetting at the moment. And I would say most as in 90% of the charts on helm hub.
Re: What we learned after a year on Kubernetes
#86I do have only very minimal xp in k8s, as I have tried to use it only once (and never again). To me it seem like mOst of the comments are about the unusable and unmaintainable yaml configuration files. This seems like exactly the issue I had a few years ago, when I was cursing, because I could not find a spec or docs for the yaml stuff in all the pages and pages of seemingly important documentation and thus could not…
You can start off with an example you would find online, then use "kubectl explain" to read about the options used and other ones you want to use.
[deleted]
Re: What we learned after a year on Kubernetes
#87Earlier quoted context omitted.
Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](…
We've found Kustomize, or just straight up writing the deployments ourselves the best approach. The actual spec for a Deployment/Daemon Set/Stateful Set/CRD is usually super straightforward once you get the Kubernetes terminology, and most of the issues I've had with Helm have boiled down to "oh they haven't parametised the one config I need to change"
I always have some configs that helm hasn't parameterized. But that's not a problem because I always fetch my charts from helm hub into my repo. So I just add any parameters I need.
Re: What we learned after a year on Kubernetes
#88I learned not to use it and wait until the k8s fad goes away.
Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…
I dunno, because of writeups like GitLabs? We're lucky GitLab is so transparent. It lets us see that their claims don't appear to always match the actual reality.
Here's an example.
Blog post says: "After transitioning each service, we enjoyed many benefits of using Kubernetes in production, including much faster and safer deploys of the application, scaling, and more efficient resource allocation"
Wrong. Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes. They tried to use auto-scaling but failed completely and ended up disabling it:
https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/920...
Kubernetes looks like a massive LOSS in the case of this service. The perceived scaling benefits didn't materialise at all, and their costs more than doubled. They also spent a lot of engineering time optimising startup time of this job: 7 PRs and complex writeups/testing were required. Just so they could try to auto-scale a job to reduce the hit of the hugely multiplied base costs. If not for Kubernetes that eng time might have been spent adding features to the product instead.
It solves really problems out out the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair
Most of these problems are created by Kubernetes, like "ingress" which is a term only Kubernetes and low level networking uses, "cert management" which often isn't required in more traditional setups beyond provisioning web servers with the SSL files, "health checks" are a feature of any load balancer and the whole point of VMs is to avoid the need to repair hardware. Finally the demand auto-scaling as seen in this case is (a) possible without Kubernetes and (b) not actually working well anyway.
Frankly this set of bugs, writeups, PRs etc is quite scary. I worked with Borg at Google and saw how it was both a force multiplier but also a huge timesink in many ways. It made some complex things simple but also made some simple things absurdly complex. At one point I had a task to just serve a static website on Borg, it turned into a three month nightmare because the infrastructure was so fragile and Google-specific. A bunch of Linux VMs running Apache would have been far faster to set up and run.
Re: What we learned after a year on Kubernetes
#89Earlier quoted context omitted.
VMs as usual.
While I agree, I'd rather deal with the k8 nonsense than maintaining infrastructure to make multiple VMs load balance, fail and restart cleanly, etc.
systemd will happily health check a service, restart it when it fails and so on. The basics don't need much.
Auto-scaling is something I'm suspicious of. I saw many experiments with that at Google when I worked there, under the name of elasticity. They found the same thing GitLab found here: it's really hard and the expected savings often don't materialise. Even in 2012 their elasticity projects were far more advanced than what Kubernetes or AWS provide.
Most cloud auto-scaling efforts appear to be driven by the high base costs charged by the cloud providers in the first place. I've seen a few cases where simply moving VMs to cheaper colo providers saves more money than auto-scaling VMs or using managed Kubernetes in AWS/Azure.
Re: What we learned after a year on Kubernetes
#90Earlier quoted context omitted.
Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…
I dunno, because of writeups like GitLabs? We're lucky GitLab is so transparent. It lets us see that their claims don't appear to always match the actual reality. Here's an example. Blog post says: "After transitioning each service, we enjoyed many benefits of using Kubernetes in production, including much faster and safer deploys of the application, scaling, and more efficient resource allocation" Wrong. Actual anal…
> Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes. They tried to use auto-scaling but failed completely and ended up disabling it:
Note this is just one service out of many and has very low latency requirements, and due to our slow pod start times in this case it didn't make sense to auto-scale. We are auto-scaling for other services in K8s.
> says their cost went from $290/month when running in VMs to $700/month
These numbers represent a snapshot in time where we are overprovisioning, for this service for the migration. You are correct that cost benefit here remains to be seen for this service. We are doing things like isolating services into separate node pools which may not allow us to be as efficient.
Safer and faster deploys was a huge win for us, for this service and others. This of course is compared to our existing configuration using VMs and Chef to manage them.
disclaimer: blog post author