231–240 of 263 posts
- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never working well - Trying to untangle Terraform state without taking down Prod
Earlier quoted context omitted.
MetalLB is pretty simply to configure.
MetalLB has been in beta for YEARS. It's OK for dev/qa/staging, but I wouldn't put in prod.
My #1 k8s nightmare is the widespread practice of just writing (or downloading and never even looking at!) YAML and applying it to the cluster, with no additional management layer (we use Terraform, but use whatever you want), meaning that eventually you have no idea what the intended state of the cluster is, only its actual state. Vendor READMEs encourage this (some even going so far as to suggest `kubectl apply -f…
>Multiple times I've found that the vendor's ...sounds about right (at least for closed source products)
Cluster-wide secret access is one of the worst I've come across, but smaller problems are virtually universal. We've come to see the YAML shipped by projects as an example, even when they document it as the preferred installation method. We always write our own now.
Even shipped Helm charts are no better, they usually encapsulate the same problems but just make them harder to fix yourself (since you are incentivised not to fork the chart as you'll have to maintain it).
It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…
Why is bare metal a nightmare? I have a project coming up which must be on bare metal so was thinking of doing this. Also, if it's so bad, what's better to use on bare metal? Thanks
You want networking? OK, go read about Calico, Flannel, Cilium, etc and choose one. If you didn't fully read the instructions for the networking plugin you plan to use, plan to blow away your cluster and set it back up from scratch with the correct RFC1918 address range for your network plugin that doesn't conflict with your presumably existing network. Plan to dive in and re-jigger things when you need IPv6.
You want a working LoadBalancer? OK, now you need MetalLB or PureLB, among others. Make sure your IPAM people know that you've blocked off several addresses or a CIDR range for K8S dynamic address allocation. IP's allocated via K8S aren't going to respond to ICMP packets and people will assume they're unused :)
You want ingress controllers? OK, well you can pick from Nginx or Traefik. There's actually a ton of them but those seem to be the most popular.
You want certificate management? OK, go install CertManager. You'll need to have programmatic access to your DNS providers if you want to use Let's Encrypt with wildcard certificates.
Oh, you need some kind of volume provider? Well.. there's hostPath but people generally don't recommend that for security reasons. I guess you could use the NFS volume provider but that's a little creaky for all of the usual reasons that NFS has been creaky for the last 30 years. You could go install Rook - but that's another entire complex distributed system ontop of your distributed system. (I love Ceph, BTW- but this is really overwhelming for a new person)
At this point you have essentially a working cluster, probably with a single master unless you set up something like OKD, in which case you already had to setup an entire HAProxy setup before even approaching the K8S parts.
Prepare to have a non-insignificant number of full time employees keeping the plane flying while you swap out the wings in real time to keep up with the fast K8S release cycle.
IMO, the complexity of K8S really incentivizes trashing all of your on-prem hardware and just paying for cloud. That's the end game.
Earlier quoted context omitted.
And don't expose workloads to the internet unless it is a prod app. Can you recommend me a good intro book to read cover to cover (hopefully not too thick).
"Kubernetes in Action" by Marko (Manning publishing) is my recommendation. Took me from someone who knows docker/docker-compose to someone who can handle Azure/AWS Kubernetes, understand the terms and design apps. Very good book.
Honestly I don't get the hate k8s get. I run my own clusters and it just works. Sure I have to ignore a lot of crap in the setup phase, there are so many products out there I don't want to pay for. The nightmare may come from some devop installing a bunch of helm charts without configuring things properly. Scaling down to a minimal cluster is a real concern: I would like to run k8s for some micro project that literal…
What about using k3s as a minimal cluster?
Ironically, I think one of the biggest issues is around packaging, specifically Helm charts (but if there are others, it is probably the same). In many frameworks, packaging is to help people by hiding complexity. Need an ingress? Use a Helm chart! But then upgrading can be very risky because if you have any problem at all, unless you understand the helm chart you can rarely simply downgrade/uninstall, you could have…
I have 2: 1. We build our own custom build system, because there is no CI that can do actual DAGs (maybe a few). A custom Kubernetes operator that parses Jsonnet files to create 100s of CRDs and pods to achieve extreme parallelization. EKS was 144$/mo (now 72$) but no info on master node types. Using watch endpoints with hundreds of pods did not scale well. They had to bump up the master node instances to c5.18xlarge…
Have you had a look at GitLab CI? They have a bit of documentation here: https://docs.gitlab.com/ee/ci/directed_acyclic_graph/
Now, I don't work on any projects that are too complicated, but I recall that piece of functionality working as one would expect: https://docs.gitlab.com/ee/ci/yaml/index.html#needs
Also there's Drone CI, which also supports setting up dependencies in your pipelines, if you'd prefer something that's not connected to GitLab CI so closely: https://docs.drone.io/pipeline/docker/syntax/parallelism/
It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…
Why is bare metal a nightmare? I have a project coming up which must be on bare metal so was thinking of doing this. Also, if it's so bad, what's better to use on bare metal? Thanks
Don't even think about using kubeadm, it's the worst. It's overcomplicated and the smallest issue will wreck your cluster.
Also as a quick tip, don't use firewalld or iptables, use CNI resources (eg calico GlobalNetworkPolicy c; )
Suppose that you work in an org that successfully ships software in a variety of ways - as regular packaged software that runs on an OS directly (e.g. a .jar that expects a certain JDK version in the VM), or maybe even uses containers sometimes, be it with Nomad, Swarm or something else.
And then a project comes along that needs Kubernetes, because someone else made that choice for you (in some orgs, it might be a requirement from the side of clients, others might want to be able to claim that their software runs on Kubernets, in other cases some dev might be padding their CV and leave) and now you need to deal with its consequences.
But here's the thing - if the organization doesn't have enough buy-in into Kubernetes, it's as if you're starting everything from 0, especially if paying some cloud vendor to give you a managed cluster isn't in the cards, be it because of data storage requirements (even for dev environments), other compliance reasons or even just corporate policy.
So, I might be given a single VM on a server, with 8 GB of RAM for launching 4 or so Java/.NET services, as that is a decent amount of resources for doing things the old way. But now, I need to fit a whole Kubernetes cluster in there, which in most configurations eats resources like there's no tomorrow. Oh, and the colleagues also don't have too much experience working with Kubernetes, so some sort of a helpful UI might be nice to have, except that the org uses RPM distros and there are no resources for an install of OpenShift on that VM.
But how much can I even do with that amount of resources, then? Well, I did manage to get K3s (a certified K8s distro by Rancher) up and running, though my hopes of connecting it with the actual Rancher tool (https://rancher.com/) to act as a good web UI didn't succeed. Mostly because of some weirdness with the cgroups support and Rancher running as a Docker container in many cases, which just kind of broke. I did get Portainer (https://www.portainer.io/) up and running instead, but back then I think there were certain problems with the UI, as it's still very much in active development and gradually receives lots of updates. I might have just gone with Kubernetes dashboard, but admittedly the whole login thing isn't quite as intuitive as the alternatives.
That said, everything kind of broke down for a bit as I needed to setup the ingress. What if you have a wildcard certificate along the lines of *.something.else.org.com and want it to be used for all of your apps? Back in the day, you'd just setup Nginx or Apache as your reverse proxy and let it worry about SSL/TLS termination. A duty which is now taken over by Kubernetes, except that by default K3s comes with Traefik as their ingress controller of choice and the documentation isn't exactly stellar.
So for getting this sort of configuration up and running, I needed to think about a HelmChartConfig for Traefik, a ConfigMap which references the secrets, a TLSStore to contain them, as well as creating the actual tls-secrets themselves with the appropriate files off of the file system, which still feels a bit odd and would probably be an utter mess to get particular certificates up and running for some other paths, as well as Let's Encrypt for other ones yet. In short, what previously would have been those very same files living on the file system and a few (dozen?) lines inside of the reverse proxy configuration, is now a distributed mess of abstractions and actions which certainly need some getting used to.
Oh, and Portainer sometimes just gets confused and fails to figure out how to properly setup the routes, though I do have to say that at least MetalLB does its job nicely.
And then? Well, we can't just ship manifests directly, we also need Helm charts! But of course, in addition to writing those and setting up the CI for packaging them, you also need something running to store them, as well as any Docker images that you want. In lieu of going through all of the red tape to set that up on shared infrastructure (which would need cleanup policies, access controls and lots of planning so things don't break for other parties using it), instead I crammed in an instance of Nexus/Artifactory/Harbor/... on that very same server, with the very same resource limits, with deadlines still looming over my head.
But that's not it, for software isn't developed in a vacuum. Throw in all of the regular issues with developing software, like not being 100% clear on each of the configuration values that the apps need (because developers are fallible, of course), changes to what they want to use, problems with DB initialization (of course, still needing an instance of PostgreSQL/MariaDB running on the very same server, which for whatever reason might get used as a shared DB) and so on.
In short, you take a process that already has pain points in most orgs and make it needlessly more complex. There are tangible benefits for using Kubernetes. Once you find a setup that works (personally, Ubuntu LTS or a similar distro, full Rancher install, maybe K3s as the underlying cluster or RKE/K3s/k0s on separate nodes, with Nginx for ingress, or a 100% separately managed ingress) then it's great and the standardization is almost like a superpower (as long as you don't go crazy with CRDs). Yet, you need to pay a certain cost up front.
What could be done to alleviate some of the pain points?
In short, I think that:
- expect to need a lot more resources than previously: always have a separate node for managing your cluster and put any sorts of tools on it as well (like Portainer/Rancher), but run your app workloads on other nodes (K3s or k0s can still be not too demanding with resources for the most part)
- don't actually shy away from tools like Portainer/Rancher/Lens for making the learning curve more shallow, inspect the YAML that they generate, familiarize yourself with the low level stuff as necessary, while still having an easy to understand overview of everything
- don't forget about needing somewhere to store Helm charts and container images, be it another node or a cloud offering of some sort
- if you can, just go for the cloud, but even if managed K8s is not in the cards for you, still strive at least for some sort of self-service approach for the inevitable reinstalls
- speaking of which, treat your clusters as *almost* disposable, have all of the instructions for preparing them somewhere, ideally as an executable script (maybe use Ansible)
- don't stray too far away from what you get out of the box, also look in the direction of the most tried and tested solutions, like an Nginx ingress (Traefik with K3s should *technically* have the better integration, but the lack of proper docs works against it, you'll probably want something like a cookbook of sorts)
- also manage your expectations, getting things up and running will probably take a long time and will be a serious aspect of development that cannot be overlooked; no, you won't have a cluster up and running on-prem with everything you need in 2 days
- ideally, have a proper DevOps team or even just a group of people who'll spearhead information sharing and creating any sorts of knowledgebases or templates so it's easier in the future