‘We couldn’t figure out k8s, so read our blog post on why it sucks’
That's really oversimplifying. EVERY tech has learning cost, and so a return on investment. K8s has been so hyped that many developers think it's a mandatory skill. It's not. I've seen talented app developers not getting through successfully deploying an app on K8s. For many, Heroku or Cloud Foundry is good enough.
Even in their case, it looks like they took a conscious decision and documented it. I'd wish more teams would do that. You are free to use K8, and I'm sure right now it's still not a controversial choice that will run against walls.
_any_ non-proprietary tool is "cloud agnostic". Kubernetes is bundled software, and achieves the things those pieces achieve. There is nothing holy about k8s specifically, the tools you train with are easy, and it's very easy to get skewed opinions on that. For example a lot of people would find writing scripts cumbersome, but not a person who's written a lot of them. They're not any more fragile than other logic err…
You didnt avtually answer the question: What free, cloud agnostic tools let me specify "keep 60% cpu load average" and work the rest out? With k8s, a horizontal autoscaler is a few lines in a yaml file and the result works in any cluster run by any vendor.
Any load balancer will let you do that. Scaling is a few lines of scripts to do on any platform, and well worth the couple minutes, you don't even need a tool for just that.
No tool "works the rest out". It's _always_ a compromise, because inherent complexity can never be removed, only moved. What you gain in one area, encumbers another.
You may freely use k8s, but it's not magical nor easier to use than any existing systems. In fact adopting it often takes non-trivial time and the web is full of failure stories with very benign warnings and catastropic results.
> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…
It's a pity that docker swarm did not make it. It wasn't perfect but it was a lot simpler to setup and manage than kubernetes. If you can get away with it, vanilla docker hosts running docker compose provide most of the same benefits with a fraction of the cost. For most startups, that's a great way to avoid getting sucked into a black hole of non value adding devops activity. You lose some flexibility but vanilla ub…
Docker Swarm doesn't support multiple users and there is no remote-accessible API. Nomad doesn't implement network policy (Nomad Connect sidecars may be an option but sidecars bring new problems). Just learn K8S, Helm, Terraform & Terragrunt properly. Use proper tools (k9s, Loki, wrapper scripts around kubectl). Stop finding excuses for not using K8S. Stop putting proxies everywhere (that Istio/servicemesh bull*hit) and use Cilium CNI instead.
> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from…
I feel like what 98% companies need is really just barebones linux with some good documentation how to spawn new nodes. To use k8s you need to know linux anyways, but to use linux you don’t need k8s knowledge. Most of things are as easy to setup with linux and the things where k8s really shines are not needed most often. What I really wonder is where did you learn k8s, which parts did you learn the most? It seems hug…
What about service discovery? What if you need to put power your node down for maintenance? What if your app goes crazy and consumes so much memory it OOMs some other services? Reliable, multi-service deployments ARE HARD and always be regardless what tech you use to achive it. Of course you can use Ansible and do it old-school way and wast lots of underutilised VMs and custom idempotent scripts. But K8S solves many of challenges in a standardised, data-driven way. It has steep learning curve but once learned, all resources/limits are properly set, logging in place, etc, it works nicely.
> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…
Nomad + Consul has been pretty great to us over the last year. We're a small nonprofit and chose it specifically because we can't afford to pay someone to keep watch over k8s
You still have to put them all together into some custom solution just for your setup which adds overhead and fragility. New employees will have to learn that instead of using K8S APIs. Deploying new components can’t take advantage of the wide and fast growing ecosystem. There’s really nothing like the full suite that kubernetes provides.
> New employees will have to learn that instead of using K8S APIs You know tech has made it when people offer argumentum ad Java. 'Sure, X might not be the best solution, but you can always buy someone to do it.'
It's also very empowering for people whose job isn't to run things, but to build the things that are to run. I've worked on a dozen-or-so of bespoke "industry standard" setups, and each and every one had a number of weird quirks, involved learning some new "industry standard" components and either made it very hard and dangerous for non-ops/devops/infra people to run their things themselves, or had homegrown tooling that pretty much replicated what the k8s API can do, just only a small subset and badly. Some YAML and kubectl are well within what a typical data scientist can be expected to understand, more so if it means they can run their things on a dev cluster themselves, and in a pinch, that data scientist can debug prod issues of their things, because it all works the same way. We have a very useful bot that was built and deployed by someone decidedly not ops while waiting for jobs to finish – a simple K8S deploymend YAML is like 50 LOC, with 40 being pretty much standard, et voilà, running bot, without having to build lots and lots of automation in-house or having to take up ops time to deploy this for them or having to grok advanced sysadmin-ing first. Used with appropriate caution and safeguards, it's super powerful.
“Plain old Linux” really isn’t an alternative to K8S though. You would need a load balancer, service discovery, a distributed scheduler, a configuration management system (K8S is a very strong alternative to building things around Ansible IMO). You can do all of those things without K8S, of course, but not with “plain old Linux” (what would that be anyway? GNU coreutils and the kernel? Vanilla Debian stable?)
Maybe I’m way off, but aren’t all of those things required for k8s? Ingress controllers, etcd cluster, terraform modules, storage configuration, etc... I guess if you pay for a hosted service a lot of the control plane is taken care of. I’ve used k8s in orgs where it’s a great fit and really fills a need, but it is considerably more complex than running a web service balanced across a couple of machines, and it defin…
I was stating that plain Linux isn’t an alternative to K8S, not that one should always use Kubernetes ;)
I work for a mid size company with 30-40 engineers managing 20-30 very diverse apps in terms of scale requirements and architectural complexity. It took our devops team (4-5 people) probably 18 months to learn and fully migrate all our apps to Kubernetes. The upfront cost was massive, but nowadays the app teams own their deployments, configurations, SLA's, monitors, and cost metrics. Introducing Kubernetes into our o…
Do you have any intuition as to what percentage of the benefits your company has seen come from kubernetes specifically, and how much just from the exercise of spending 18 months working on a modern, coherent, efficient infrastructure?
Kubernetes is almost a standard, all our engineers are eager to learn it. It is extensible, some app teams have started building operators to tackle problems in a more efficient way. It allowed us to standardize metrics and monitoring, we didn’t have a clear story for this before. It is cloud agnostic and with great compatibility; I was part of a migration from GKE to EKS, and it was painless.