Live data from Hacker News

Kubernetes is a red flag signalling premature optimisation

jeremybrown.tech

171–180 of 558 posts

Re: Kubernetes is a red flag signalling premature optimisation

#171
post #112

Earlier quoted context omitted.

It's really not unless you are doing a half-assed job of setting up bare boxes. Lets see to get something even half-reasonable on a bare box you need the minimum: process monitor: For this you can use systemd, supervisord, etc. logging: rsyslog or similar. http reverse proxy: nginx or haproxy deployment mechansim: probably scp if you are going this ghetto but git pull + build and/or pull from s3 are all common at thi…

I mean, container+registry is possibly more complex to start and maintain than just creating a service file with SystemD. Deployment can be as easy of "systemctl restart service", and seems to me that configuring all of those resources in Kubernetes is far more difficult that just setting up a simple service on a bare box. Not to mention that you could use Docker too. And by the looks of it, ingress doesn't seem triv…

You don't run your own registry (except in very rare circumstances where that makes sense). You author Dockerfile (or better yet use a tool like jib that creates containers automatically without even a Docker daemon), then you push to hosted registry.

Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you think whatever you are doing is very special it probably does it already. That is the benefit of literal thousands of teams using the exact same abstraction layer.

You don't have SSH (this is objectively good), instead you can get exec on a container if you need it (and that container contains a shell). You can now control on a fine grained and integrated fashion exactly who is allowed to exec into a container and you get k8s audit events for free. (and if you are using a hosted system like GKE then it automatically flows into said providers audit system).

The point is you can buy managed k8s, there isn't an equivalent for old school sysadmin that doesn't amount to outsourcing to a body shop.

Given that you haven't managed to identify any of k8s real downsides here they are:

It's a bunch of yaml. Yeah I don't like that either but there are tools like Tanka that make that suck less.

It's complex under the hood. When your process runs its in a network, fs and pid namespaces and you have cgroup managing resources. There is probably some sort of overlay network or some other means of granting each container an IP and making that routable (BGP, OSPF, etc). Interaction between ingress, service and pods is through an indirection layer called endpoints that most people don't even realize exists. Scheduling is complex, there are tunables for affinity, anti-affinity, it also needs to interact with cluster autoscaling. Autoscaling itself is a complex topic where things like hot capacity etc aren't exactly sorted out yet. QoS, i.e pod priority isn't well understood by most. When will a pod be pre-empted? What is the difference between eviction/pre-emption etc? Most people won't be able to tell you. This means when things go wrong there are a lot of things that could be wrong, thankfully if you are paying someone else for your cluster a) it's their problem b) it's probably happening to -all- of their customers so they have a lot of incentive to fix it for you.

Multi-cluster is a mess, no reasonable federation options in sight. Overlay networking makes managing network architecture when involving multiple clusters more difficult (or external services outside of said cluster).

Ecosystem has some poor quality solutions. Namely helm, kustomize, Pulumi, etc. Hopefully these will die out some day and make way for better solutions.

Yet for all of these downsides it's still clearly a ton better than managing your own boxes manually. Especially because you are -exceedingly- unlikely to encounter any of these problems above at any scale where self-hosting would have been tennable.

I think the best way to think about k8s is it's the modern distributed kernel. Just like Linux you aren't expected to understand every layer of it, merely the interface (i.e resource API for k8s, syscalls/ioctl/dev/proc/sysfs for Linux). The fact everyone is using it is what grants it the stability necessary to obviate the need for that internal knowledge.

Re: Kubernetes is a red flag signalling premature optimisation

#172
I'm not sure if I can agree with the author. We moved from plain AWS EC2 to K8 to improve our continuous delivery pipeline and it worked. Before we had dozen of custom terraform scripts, custom runners on gitlab, and it became much cleaner now with K8. Other aspect that improved was monitoring and introspection. I think mainly because K8 offers a reach set of tools and patterns helping exactly in that matters. It comes with its own costs, but I can definitely say that we went from a tailor made solution to a standard set of techniques and tools, which is a huge improvement, IMO.

Re: Kubernetes is a red flag signalling premature optimisation

#174
post #6

Earlier quoted context omitted.

Doesn't it depend what staff you have? I would have agreed with this in 2018 but the world has moved on. If you have a bunch of people who know it, you can deploy a cluster into gcloud or aws with a few clicks or lines of IaC. Would I recommend a startup team learn kube while trying to ship a product? No. Would I think it's a red flag if a team who already know it choose it as their preferred platform? Also no.

Even if they know it, unless they absolutely need to implement it, why would you waste such valuable resources on of all things infra? (Unless your product IS infra). No engineer I know who’s smart enough to effortlessly deploy k8s on their own would want to do that as the job. There’s a million other interesting things (hopefully?) that they can be doing.

Often to get access to things that integrate well with kubernetes.

Personally I don't see too much difference between kube yamls and systemd units and cloudformation or whatever and the "cluster maintenance" itself is not the burden it used to be if you stay on the "paved road" provided by your cloud.

Re: Kubernetes is a red flag signalling premature optimisation

#175

Doesn't look like the author knows what he is talking about. His point about early stage startup should not use K8S is fine. But the next advice about not using a different language for frontend and backend is wrong. I think the most appropriate advice is to choose a stack which the founding team is most familiar with. If that means RoR then RoR is fine. If it means PHP then PHP is fine too. Another option is to use…

I don't think their advice about not using it in a startup is correct either. You just need to somewhat know what you're doing. I know of such a case, where a single engineer could leverage the helm chart open source community, and set up a scalable infrastructure, with prometheus, grafana, worker nodes that can scale independently of web service, a CI/CD pipeline that can spin up complete stacks with TLS automated t…

How long did it take him to do this setup, a year you say, and that is impressive? I am not trying to be cute here, my question comes from a genuine place of curiosity. I've love to learn to spin-up a system like that, but from the tech/sales talks I see I am made to believe this can be done in a day. Expectation management is important, if people say ops is just a solved problem then I expect this to take very little time and to be easy to learn. Maybe I am learning the wrong thing here, and should do learn Helm or something more high level.

Re: Kubernetes is a red flag signalling premature optimisation

#176
post #71
post #42

Earlier quoted context omitted.

I think the argument was/is not "its a problem we cannot have javascript" but "if we can have javascript everywhere, we only need to hire javascript devs and only need to care about javascript tooling", which is a fair point. That does ignore the fact that an experienced frontend JS dev is not necessarily also a good productive backend JS dev, but at least they know the language basics, can use the same IDE, etc. If…

I think this has been a huge failing of our industry of late. The rise of the "fullstack developer" has mostly reduced quality across the board. When you hire a "fullstack developer with 5 years experience" you aren't getting someone who is as good as a frontend developer with 5 years AND a backend developer with 5 years but someone that adds up to 5 years split between those 2 endeavors but probably with less depth…

Most 'fullstack' positions don't have nearly the complexity where worrying about concurrency etc. is actually that relevant. The idea that most frontend devs have any knowledge about optimising for browser paint performance beyond using the correct CSS or framework is funny ;-)

Re: Kubernetes is a red flag signalling premature optimisation

#177
I'll be honest - i've worked places where kubernetes was a thing, and places where it wasn't. Both within the last 5 years.

Kubernetes is a layer of complexity that just isn't warranted for most companies. Hell even Amazon still sticks with VMs. Autoscaling and firecracker solve most things.

Its nice to have cluster (pod?) management as a first-class concept, but you get that with tagging of instances just as well for most use cases.

In short - i think the author has a point.

Re: Kubernetes is a red flag signalling premature optimisation

#178

Doesn't look like the author knows what he is talking about. His point about early stage startup should not use K8S is fine. But the next advice about not using a different language for frontend and backend is wrong. I think the most appropriate advice is to choose a stack which the founding team is most familiar with. If that means RoR then RoR is fine. If it means PHP then PHP is fine too. Another option is to use…

The Whole advice of using same language is especially silly - iOS is stuck with Swift, and the web is stuck with JS, and maybe you need an applitation that scales using actors across mutiple machines with Golang or Java, or maybe you need to plug into Windows tightly and need C#.

Kubernetes is not 'harder' if all you need is to host a webapp. Where it falls on the hardness spectrum depends on what you are trying to do, and what is the alternative. I am very fluent with Kubernetes but have no skills in managing traditional virtual machines.

Re: Kubernetes is a red flag signalling premature optimisation

#179

I'm a developer that uses Kubernetes in production purely because I want to be able to use the same Docker images that I use in development. I am not a Kubernetes advocate but what else is there that handles all of the issues faced when deploying containers? Such as scaling, deployment, configuration etc? There are alternatives such as Hashicorps Nomad, but I don't see how this is any better/worse that K8s.

Check into Amazon ECS or Google Cloud Run.

It's basically a higher level of managed Kubernetes ( EKS or GKE ). You give your containers, and they take care of running it.

Tim Hockin (one of kubernetes creator) support the idea to use something as much managed and automatic as possible : https://twitter.com/thockin/status/1539987108521054208

Re: Kubernetes is a red flag signalling premature optimisation

#180
As a startup founder that's not VC funded, I would totally recommend you look into building with kubernetes from the get go. The biggest learning curve is for the person setting up the initial deployments, services, ingress etc Most other team members may just need to maybe change the image name and kubectl apply to roll things out. Knowing that rollouts won't bring down prod and that they can be tested in different environments consistently is really valuable.

I started out with Iaas namely Google App Engine and we suffered a ton with huge bills especially from our managed db instance. Once the costs were too high we moved to VMs. Doing deployments was fine but complicated enough that only seasoned team members could do it safely. We needed to build a lot of checks, monitoring etc to do this safely. A bunch of random scripts existed to set things up and migrating base operating system etc required a ton of time. Moving to kubernetes was a breath of fresh air and I wish we'd done it earlier. We now have an easy repeatable process . Infra is easier to understand. Rollouts are safer and honestly, the system is safer too. We know exactly what ports can allow ingress, what service boundaries exist. What cronjobs are configured, their state etc with simple kubectl commands.

Using kubernetes forces you to write configurable code and is very similar to testing: it sounds like it'll slow you down and shouldn't be invested in until the codebase is at a certain size but we've all learned from experience how is actually speeds everything up, makes larger changes faster, cheaper customer support and saves you from explaining why a certain feature has been broken for 10 without anyone's knowledge

Post reply on HN