Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Kubernetes is so complex

cloudplane.org

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Why Kubernetes is so complex

#41

Post by thunky »

marginalia_nu wrote: post #35
xyzzy123 wrote: post #32

Earlier quoted context omitted.

Mostly agree, I prefer k8s to learning the custom duct tape for every project. Observation: a side effect of being extensible is that people deploy extensions. There is some kind of law of complexity budgets, where if you make the simple things easy, people will tend to ratchet up complexity by adding more stuff until the system "just" fits in their heads again. Bare k8s with a simple ingress path and workload is pre…

You kinda need to still learn the custom duct tape though. Kubernetes is something you need to learn on top of all the details. It doesn't replace it. (It may however seem that way until you run into any actual problems with your software)

Yeah it's more like a network of duct tape webbed together within in a clear plastic box with a "kubectl" button on it along with a paper feeder that takes a stack of generated config files as input for each button press.

And this controls how the duct tape sticks everything together, so it's not like any two boxes are the same.

Re: Why Kubernetes is so complex

#42

Post by speed_spread »

perfecthjrjth wrote: post #28
vaidhy wrote: post #4

I think the answer is hidden in the last paragraph. It was created by Google engineers to solve Google scale problems where they run maybe hunderds of thousands of services with a dedicated SRE team and massive tooling. For a small company, give me a way to run one container in an autoscable way and we can go from there.

Large companies are done with huge k8s clusters. Now every app team and their SRE team rolls out their own k8s cluster. I am not sure whether these companies are solving Google scale problems.

Maybe they weren't solving Google-scale problems before but with Kubernetes they sure are now!

Re: Why Kubernetes is so complex

#44

Post by schmichael »

cb22 wrote: post #29
techdragon wrote: post #25

Earlier quoted context omitted.

I recently looked at Nomad, but ironically for a "less complex" piece of software, I couldn't be sure it would scale down to my use case, I ended up with K3S as it would run on the small hardware nodes i needed it to run on. I would have preferred Nomad but the resource requirements are pretty high for the "control server" component. https://www.nomadproject.io/docs/install/production/requirem... Nomad servers may ne…

I too was initially put off by those requirements. Now we run our Nomad server on a single t3.medium instance that sits at a 0.1 15 minute load average and has ~500MB of RAM used. This manages about 100 client nodes. No need for a cluster since we don't need high availability on our control plane, and there's no actual state stored there that isn't created from our CI pipeline.

Glad you went for it despite our poor documentation cb22! That sounds like a great setup. I think the only defensible way to describe our "Requirements" page is that we wanted to make the safest suggestion for the widest range of users. Obviously it's wildly inaccurate for a wide range use cases, and we should fix that.

Nomad servers could start with 300mhz, 100mb of memory, and eMMC storage and run a RaspberryPI cluster just fine. Our most important resource guidance is all the way over in our Monitoring docs!

> Nomad servers' memory, CPU, disk, and network usage all scales linearly with cluster size and scheduling throughput.

https://developer.hashicorp.com/nomad/docs/operations/monito...

Any cluster can start with 300mhz and 100mb as long as they monitor usage and scale appropriately.

We're going to try to update our Requirements docs to add this nuance and guidance on how to calculate requirements based on projected load. We recently spent some time improving our heartbeat docs, and I think the approach we took there will serve us well for system requirements: https://developer.hashicorp.com/nomad/docs/configuration/ser...

Re: Why Kubernetes is so complex

#45

Post by mnomitch »

Cwizard wrote: post #26

Could someone explain to me in what why Kubernetes is complex and what alternatives are simpler? I’ve worked on non-k8s systems before and in my experience they all hang together with custom bash/python code which although line-for-line is ‘simpler’ it makes it harder to onboard new people and is less robust (excluding very simple deployments) K8s is very modular in my experience so if you don’t need something you ca…

Nomad does not seem much simpler to me (especially because you basically have to pair it with Consul and Vault)

Hi, Nomad PM here - We've gotten this feedback a lot and have been taking steps to respond to it. We added simple service discovery in Nomad 1.3 and health checks and load balancing shortly after. So you shouldn't need Consul until you want a full service mesh. And then in Nomad 1.4, which just launched, we added Nomad Variables. These can be used for basic secrets & config management. It isn't a full on replacement for Vault, but it should give people the basic functionality to get off the ground.

So going forward we won't have a de factor dependency on these other tools, and hopefully we can live up to the promise of simplicity.

Re: Why Kubernetes is so complex

#46

Post by jbergens »

MuffinFlavored wrote: post #5
vaidhy wrote: post #4

I think the answer is hidden in the last paragraph. It was created by Google engineers to solve Google scale problems where they run maybe hunderds of thousands of services with a dedicated SRE team and massive tooling. For a small company, give me a way to run one container in an autoscable way and we can go from there.

> For a small company, give me a way to run one container in an autoscable way and we can go from there. What's the best way you've found to do that?

Azure is really simple if you use dotnet. Just create an app and then click scale-out (or scale-up).

They of course support docker and k8 also. And Azure functions which are like lambda in AWS.

Re: Why Kubernetes is so complex

#47

Post by techdragon »

schmichael wrote: post #44
cb22 wrote: post #29

Earlier quoted context omitted.

I too was initially put off by those requirements. Now we run our Nomad server on a single t3.medium instance that sits at a 0.1 15 minute load average and has ~500MB of RAM used. This manages about 100 client nodes. No need for a cluster since we don't need high availability on our control plane, and there's no actual state stored there that isn't created from our CI pipeline.

Glad you went for it despite our poor documentation cb22! That sounds like a great setup. I think the only defensible way to describe our "Requirements" page is that we wanted to make the safest suggestion for the widest range of users. Obviously it's wildly inaccurate for a wide range use cases, and we should fix that. Nomad servers could start with 300mhz, 100mb of memory, and eMMC storage and run a RaspberryPI clu…

Thank you for the follow up! This makes it much more likely that I’ll remember to give it a shot when the next applicable project comes up… Because I’ll probably have forgotten these comments and gone to check the docs again :-)
Post reply on HN