Preach, however all fucking job ads want proficiency in k8s.
So screwed if you do, screwed if you don't
I Didn't Need Kubernetes, and You Probably Don't Either
221–230 of 436 posts
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#222I’ve come to the conclusion that I hate “cloud shit”, and a small part of me is convinced that literally no one actually likes it, and everyone is playing a joke on me. I have set up about a dozen rack mount servers in my life, installing basically every flavor of Unix and Linux and message busses under the sun in the process, but I still get confused by all the Kubectl commands and GCP integration with it. I might j…
Isn't using VMs or servers just cloud shit in a different form? How does that fix anything? Orchestrating software (or hardware) is a pain in the ass whether it's in your closet or someone else's.
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#223I’ve come to the conclusion that I hate “cloud shit”, and a small part of me is convinced that literally no one actually likes it, and everyone is playing a joke on me. I have set up about a dozen rack mount servers in my life, installing basically every flavor of Unix and Linux and message busses under the sun in the process, but I still get confused by all the Kubectl commands and GCP integration with it. I might j…
I have pretty much the exact same opinion, right down to “cloud shit” (I have even used that exact term at work to multiple levels of management, specifically “I’m not working on cloud shit, you know what I like working on and I’ll do pretty much anything else, but no cloud shit”). I have worked on too many projects where all you need is a 4-8 vCPU VM running nginx and some external database, but for some reasons the…
Speaking of Kubernetes performance: I had a need for fast scale-out for a bulk testing exercise. The gist of it was that I had to run Selenium tests with six different browsers against something like 13,000 sites in a hurry for a state government. I tried Kubernetes, because there's a distributed Selenium runner for it that can spin up different browsers in individual pods, even running Windows and Linux at the same time! Very cool.
Except...
Performance was woeful. It took forever to spin up the pods, but even once things had warmed up everything just ran in slow motion. Data was slow to collect (single-digit kilobits!), and I even saw a few timeout failures within the cluster.
I gave up and simply provisioned a 120 vCPU / 600 GB memory cloud server with spot pricing for $2/hour and ran everything locally with scripts. I ended up scanning a decent chunk of my country's internet in 15 minutes. I was genuinely worried that I'd get put on some sort of "list" for "attacking" government sites. I even randomized the read order to avoid hammering any one site too hard.
Kubernetes sounds "fun to tinker with", but it's actually a productivity vampire that sucks up engineer time.
It's the Factorio of cloud hosting.
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#224I don't get these recent anti-Kubernetes posts, yes if you're deploying a simple app then there are alternatives which are easier, but as your app starts to get more complex then suddenly you'll be wishing you had the Kubernetes API. I'd use Kubernetes even if I was spinning up a single VM and installing k3s on it. It's a universal deployment target. Spinning up a cluster isn't the easiest thing, but I don't understa…
lol, even if you have complex apps there are always easier solutions than Kubernetes. It is evident that you have never run such an app and are just talking about it. Otherwise, you would know the issues you would encounter with every update due to breaking changes. Not to mention that you need a high level of expertise and a dedicated team, which costs far more than running an app on Fargate. Recommending a managed…
I've been running apps on Kubernetes clusters for the past 6 years and the only thing that really comes to mind that was a breaking change was when the ingress class resource type was introduced. Everything else has been incremental. Maybe I'm forgetting something.
What's wrong with recommending a managed cluster? I wouldn't use one but it is certainly an option for teams that don't want to spin up a cluster from scratch, although it comes with its own set of tradeoffs.
My project at the moment is definitely easier thanks to Kubernetes as pods are spun up dynamically and I've migrated to a different cloud provider and since migrated to a mix of dedicated servers and autoscaled VMs, all of which was easy due to the common deployment target rather than building on top of a cloud provider specific service.
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#225People talk about Kubernetes as container orchestration, but I think that's kind of backwards. Kubernetes is a tool for creating computer clusters. Hence the name "Borg" (Kubernetes's grandpa) referring to assimilating heterogeneous hardware into a collective entity. Containers are an implementation detail. Do you need a computer cluster? If so k8s is pretty great. If you don't care about redundancy and can get all t…
> Containers are an implementation detail.
They really aren't.
Personally I have a big Nix derivation to deploy my (heterogeneous) cluster to bare metal.
None of the k8s concepts or ideas apply here.
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#226> Kubernetes comes with substantial infrastructure costs that go beyond DevOps and management time. The high cost arises from needing to provision a bare-bones cluster with redundant management nodes. That's your problem right there. You really don't want to be setting up and managing a cluster from scratch for anything less than a datacenter-scale operation. If you are already on a cloud provider just use their mana…
Even for on-prem scenario, I'd rather maintain a K8S control plane and let developer teams manage their own apps deployment in their own little namespace, than provisioning a bunch of new VMs each time a team need some services deployed.
This for me is THE reason for using container management. Without containers, you end up with hundreds of VMs. Then, when the time comes that you have to upgrade to a new OS, you have to go through the dance, for every service:
- set up new VMs
- deploy software on new VMs
- have the team responsible give their ok
It takes forever, and in my experience, often never completes because some snowflake exists somewhere, or something needs a lib that doesn't exist on the new OS. VMs decouple the OS from the hardware, but you should still decouple the service from the OS. So that means containers. But then managing hundreds of containers still sucks.
With container management, I just
- add x new nodes to cluster
- drain x old nodes and delete them
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#227Earlier quoted context omitted.
I agree, but what pisses me the most is that today higher level abstractions (like cloud, spring boost,...) are hiding lower level functionality so well, that you are literally forced to use obnoxious amounts of time to study documentation (if you are in luck and it is well written), while everything is decorated with new naming of known concepts that was invented by people who didn't know that the concept already ex…
The real benefits of LLMs is to give me a good summary of these documents. Still, using these abstractions is probably not worth it.
Well, a summary. It's impossible to evaluate the quality of the summary without actually reading the document you're trying to avoid reading.
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#228> Kubernetes comes with substantial infrastructure costs that go beyond DevOps and management time. The high cost arises from needing to provision a bare-bones cluster with redundant management nodes. That's your problem right there. You really don't want to be setting up and managing a cluster from scratch for anything less than a datacenter-scale operation. If you are already on a cloud provider just use their mana…
For most small setups, the cost of running an empty kubernetes cluster (managed) is typically higher than setting up a db, a couple of vms and a loadbalancer, which goes a long way for running a simple service. Add some buckets, a CDN and you are pretty much good to go. If you need dedicated people just to stay on top of running your services, you have a problem that's costing you hundreds of thousands per year. Ther…
> For most small setups, the cost of running an empty kubernetes cluster (managed) is typically higher than setting up a db, a couple of vms and a loadbalancer, which goes a long way for running a simple service.
Not really, the cost of an empty EKS cluster is the management fee of $0.1/hour, or roughly the price of a small EC2 instance.
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#229Earlier quoted context omitted.
Or if you got a static ip and fast up speed, then just port forward 80, 443 and start hosting your self. Even an old Intel Mac Book pro from 2000's with 4 GB of RAM may not be that hot running MacOS, but install Debian with no X. It is running smooth as a whistle, while running several conduit matrix, haraka, zone-mta, ice cast, nginx with no issues. WebRTC/TUN/STUN becomes an issue with the nginx config. May conside…
> It is running smooth as a whistle ... until you get hit by a DDoS attack. Not much you can do about it unless your ISP offers protection, or you end up going for Cloudflare or the like instead of exposing your IP and ports.
Folding the second you get dos'd is aa feature, not a liability. Staying up through those is a great way to lose a ton of money basically instantly.
Re: I Didn't Need Kubernetes, and You Probably Don't Either
#230Earlier quoted context omitted.
I'm always kind of blown away by experiences like this. Admittedly, I've been using Kubernetes since the early days and I manage an Infra team that operates a couple thousand self-managed Kubernetes clusters so... expert blindness at work. Before that I did everything from golden images to pushing changes via rsync and kicking a script to deploy. Maybe it's because I adopted early and have grown with the technology i…
Those "necessary" add-ons and sidecars are out of control, but its the people problem. I'm part of the infra team and we manage just couple of k8s clusters, but those are quite big and have very high traffic load. The k8s + terraform code is simple, with no hacks, reliable and easy to upgrade. Our devs love it, we love it too and all of this makes my job pleasant and as little stressful as possible. But we recently h…
How do you scale mTLS ops when the CISO comes knocking?