I Didn't Need Kubernetes, and You Probably Don't Either • Hacker News Board

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#331

Post by immibis » Wed, Nov 27, 2024, 1:58 PM UTC

Havoc wrote: post #295

tombert wrote: post #90

I’ve come to the conclusion that I hate “cloud shit”, and a small part of me is convinced that literally no one actually likes it, and everyone is playing a joke on me. I have set up about a dozen rack mount servers in my life, installing basically every flavor of Unix and Linux and message busses under the sun in the process, but I still get confused by all the Kubectl commands and GCP integration with it. I might j…

“let me install everything myself” doesn’t generalise well and gets messy even if you IaC it. There is a reason k8s gang keeps going on about “cattle not pets”. The starting assumptions and goals are fundamentally different vs “give me a physical server” Both have their place I think so not really one is right other is wrong

And people tend to vastly underestimate the power of a single server. These days you can get a terabyte of RAM, 96 cores and dual 10G Ethernet for a low-to-mid 4-digit price (used). Do you need the cloud? Some do, but often your highest conceivable scale fits on one or two servers. Stack Exchange famously ran this way for a long time (until recently when they've been bought by large investors and they're going full cloud and AIshit).

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#332

Post by voidfunc » Wed, Nov 27, 2024, 2:01 PM UTC

quietbritishjim wrote: post #233

voidfunc wrote: post #107

Earlier quoted context omitted.

I'm always kind of blown away by experiences like this. Admittedly, I've been using Kubernetes since the early days and I manage an Infra team that operates a couple thousand self-managed Kubernetes clusters so... expert blindness at work. Before that I did everything from golden images to pushing changes via rsync and kicking a script to deploy. Maybe it's because I adopted early and have grown with the technology i…

I hardly know the first thing about Kubernetes or cloud, so maybe you can help explain something to me: There's another Kubernetes post on the front page of HN at the moment, where they complain it's too complex and they had to stop using it. The comments are really laying into the article author because they used almost 50 clusters. Of course they were having trouble, the comments say, if you introduce that much com…

> But then here you are saying your team "operates a couple thousand" clusters. If 50 is far too many, and bound to be unmanageable, how is it reasonable to have more than a thousand?

It's not unmanageable to have a couple thousand Kube clusters but you need to have the resources to build a staff and tool chain to support that, which most companies cannot do.

Clusters are how we shard our customer workloads (a workload being say a dozen services and a database, a customer may have many workloads spread across the entire fleet). We put between 100 and 150 workloads per cluster. What this gives us is a relatively small impact area if a single cluster becomes problematic as it only impacts the workloads on it.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#333

Post by p_l » Wed, Nov 27, 2024, 2:06 PM UTC

Spivak wrote: post #65

ants_everywhere wrote: post #42

People talk about Kubernetes as container orchestration, but I think that's kind of backwards. Kubernetes is a tool for creating computer clusters. Hence the name "Borg" (Kubernetes's grandpa) referring to assimilating heterogeneous hardware into a collective entity. Containers are an implementation detail. Do you need a computer cluster? If so k8s is pretty great. If you don't care about redundancy and can get all t…

This has got to be the most out there k8s take I've read in a while. k8s doesn't save you from learning your cloud providers infrastructure, you have to learn k8s in addition to your cloud provider's infrastructure. It's all ALBs, ASGs, Security Groups, EBS Volumbes and IAM policy underneath and k8s, while very clever, isn't so clever as to abstract much of any of it away from you. On EKS you get to enjoy more odd li…

K8s was designed around deployment on premise on bare metal hardware.

The cloud extensions were always just a convenience.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#334

Post by zelphirkalt » Wed, Nov 27, 2024, 2:07 PM UTC

tombert wrote: post #90

I’ve come to the conclusion that I hate “cloud shit”, and a small part of me is convinced that literally no one actually likes it, and everyone is playing a joke on me. I have set up about a dozen rack mount servers in my life, installing basically every flavor of Unix and Linux and message busses under the sun in the process, but I still get confused by all the Kubectl commands and GCP integration with it. I might j…

I am sure the consultants and others peddling Kubernetes like it, since they are getting the paycheck for bringing it on, or cementing their position, as they are the ones with the expert knowledge of the system then, if anyone is at all, after something has been migrated to Kubernetes.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#335

Post by p_l » Wed, Nov 27, 2024, 2:08 PM UTC

nicce wrote: post #74

sobellian wrote: post #27

Earlier quoted context omitted.

I've worked at a few tiny startups, and I've both manually administered a single server and run small k8s clusters. k8s is way easier. I think I've spent 1, maybe 2 hours on devops this year. It's not a full-time job, it's not a part-time job, it's not even an unpaid internship. Perhaps at a bigger company with more resources and odd requirements...

But how much this costs extra? Sounds like you are using cloud-provided k8s.

I currently use k8s to control bunch of servers.

The amount of work/cost of using k8s for handling them in comparison to doing it "old style" is probably negative by now.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#336

Post by p_l » Wed, Nov 27, 2024, 2:20 PM UTC

kccqzy wrote: post #51

valenterry wrote: post #12

Earlier quoted context omitted.

So, let's say you want to deploy server instances. Let's keep it simple and say you want to have 2 instances running. You want to have zero-downtime-deployment. And you want to have these 2 instances be able to access configuration (that contains secrets). You want load balancing, with the option to integrate an external load balancer. And, last, you want to be able to run this setup both locally and also on at least…

I've worked at tiny startups before. Tiny startups don't need zero-downtime-deployment. They don't have enough traffic to need load balancing. Especially when you are running locally, you don't need any of these.

Tiny startups don't have money to spend on too much PaaS or too many VMs or faff around with custom scripts for all sorts of work.

Admittedly, if you don't know k8s, it might be non-starter... but if you some knowledge, k3s plus cheap server is a wonderful combo

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#337

Post by p_l » Wed, Nov 27, 2024, 2:22 PM UTC

pclmulqdq wrote: post #329

worldsayshi wrote: post #164

Earlier quoted context omitted.

Yeah this is my impression as well which makes me not understand the k8s hate.

The complexity of k8s comes the moment you need to hold state of some kind. Now instead of one systemd entry, we have to worry about persistent volume claims and other such nonsense. When you are doing things that are completely stateless, it's simpler than systemd.

If you need to care about state with systemd you still have the "nonsense" of persistent volume claims, they are just something you keep in notes somewhere, in my experience usually in heads of the sysadmins or an excel sheet or a text file that tries to track which server has what data connected how.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#338

Post by p_l » Wed, Nov 27, 2024, 2:23 PM UTC

solatic wrote: post #81

Cloud Run is fine if you're a small startup and you're thinking about your monthly bill in three-figure or even four-figure terms. Like most serverless solutions, it does not permit you to control egress traffic. There are no firewall controls exposed to you, so you can't configure something along the lines of "I know my service needs to connect to a database, that's permitted, all other egress attempts are forbidden…

kubernetes is how I keep compute costs in 2-3 digits :V

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#339

Post by p_l » Wed, Nov 27, 2024, 2:25 PM UTC

rnts08 wrote: post #216

The worst thing about the whole kubernetes-cult and the like is that IaC and CM was supposed to help us reduce configuration, to make it less prone to failure and easier to manage. But the truth is that we ran service based architecture, network meshes and containers with bash just fine before cloud everything, usually with less effort than it is to do literally anything today. Sure you had to know how to set up netw…

As I hope decent system administrator... turns out I find running minimal kubernetes setup (k3s is great there) more sensible and cheaper than managing the server in classic way.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#340

Post by p_l » Wed, Nov 27, 2024, 2:36 PM UTC

t-writescode wrote: post #108

threeseed wrote: post #19

Earlier quoted context omitted.

Kubernetes has a proportional learning curve. If you're used to managing platforms e.g. networking, load balancers, security etc. then it's intuitive and easy. If you're used to everything being managed for you then it will feel steep.

I think this is only true if the original k8s cluster you're operating against was written by an expert and laid out as such. If you're entering into k8s land with someone else's very complicated mess across hundreds of files, you're going to be in for a bad time. A big problem, I feel, is that if you don't have an expert design the k8s system from the start, it's just going to be a horrible time; and, many people, w…

Thanks to kubernetes "flattening" that mess into somewhat flat object map (I like to call it a blackboard system :D) it can be reasonably easy to figure out what's the desired state and what's the current state for given cluster, even if the files are a mess.

However...

Talking with people who started using kubernetes later than me[1], it seems like a lot of confusion starts by trying to start with somewhat complete example like using a Deployment + Ingress + Services to deploy, well, a typical web application. The stuff that would be trivial to run in typical PaaS.

The problem is that then you do not know what a lot of those magic incantations mean, and the actually very, very simple mechanism of how things work in k8s are lost, and you can't find your way in a running cluster.

[1] I started learning around 1.0, went with dev deployment with 1.3, graduated it to prod with 1.4. Smooth sailing since[2]

[2] The worst issues since involved dealing with what was actually global GCP networking outage that we were extra sensitive to due to extensive DNS use in kubernetes, and once naively assuming that the people before me set sensible sizes for various nodes, only to find a combination of too small to live EC2 instances choking till control plane death, and outdated etcd (because the rest of the company twas too conservative in updating) getting into rare but possible bug that corrupted data which was triggered by the flapping caused by too small instances. Neither I count as k8s issue, would have killed anything else I could setup given the same constraints.