51–60 of 79 posts
I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.
I was of the same opinion, so I rolled my own thin layer over Compose: https://harbormaster.readthedocs.io/
Earlier quoted context omitted.
A pithy response to be sure, but is it true? Every Kubernetes object type exists within a well-specified hierarchy, has a well-specified specification, an API version, and documentation. Most of the object families' evolution are managed by a formal SIG. Not sure how any of that qualifies as ad-hoc or informal.
"It's not a mess! it was designed by committee!" I'm not sure what to say here. The kubernetes docs and code speak for themselves. If you actually think that it's clean, simple, well designed, and easy to operate, with smooth interop between the parts, I can't change your mind. But in practice, I have found it very unpleasant. It seems this is common, and the usual suggestion is to pay someone else to operate it.
Also I never said Kubernetes was well-designed, easy, or simple.
I recently rebuilt my Kubernetes cluster running across three dedicated servers hosted by Hetzner and decided to document the process. It turned into a (so far) 8-part series covering everything from bootstrapping and firewalls to setting up persistent storage with Ceph. Part I: Talos on Hetzner https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talo... Part II: Cilium CNI & Firewalls https://datavirke.dk/posts/…
Have you tried KubeOne? Also with the benefits of machine-deployments. Works like a charm, didn’t go through your blogs, but KubeOne on Hetzner [0] seems easier than your deployment. And yes, also Open Source and German support available. [0] https://docs.kubermatic.com/kubeone/main/architecture/suppor...
I haven't used KubeOne, but I have previously used Syself's https://github.com/syself/cluster-api-provider-hetzner which I believe works in a similar fashion. I think the approach is very interesting and plays right into the Kubernetes Operator playbook and its self-healing ambitions.
That being said, the complexity of the approach, probably in trying to span and resolve inconsistencies across such a wide landscape of providers, caused me quite a bit of grief. I eventually abandoned this approach after having some operator somewhere consistently attempt and fail to spin up a secondary control plane VPS against my wishes. After poring over loads of documentation and half a dozen CRDs in an attempt to resolve it, I threw in my hat.
Of course, Kubermatic is not Syself, and this was about a year ago, so it is entirely possible that both projects are absolutely superb solutions to the problem at this point.
Earlier quoted context omitted.
I was of the same opinion, so I rolled my own thin layer over Compose: https://harbormaster.readthedocs.io/
This looks really nice, but the main feature of Docker Swarm rather than, Docker Compose, is the ability to run on a cluster of servers, not just a single node.
I recently rebuilt my Kubernetes cluster running across three dedicated servers hosted by Hetzner and decided to document the process. It turned into a (so far) 8-part series covering everything from bootstrapping and firewalls to setting up persistent storage with Ceph. Part I: Talos on Hetzner https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talo... Part II: Cilium CNI & Firewalls https://datavirke.dk/posts/…
Interesting read. I have just setup a very similar cluster this week: 3 node bare metal cluster in a 10G mesh network. Decided for Debian, RKE2, Calico and Longhorn. Encryption is done using LUKS FDE. For Load Balancing I am using the HCloud Load Balancer (in TCP mode). At first I had some problems with the mesh network as the CNI would only bind to a single interface. Finally solved it using a bridge, veth and isola…
I recently rebuilt my Kubernetes cluster running across three dedicated servers hosted by Hetzner and decided to document the process. It turned into a (so far) 8-part series covering everything from bootstrapping and firewalls to setting up persistent storage with Ceph. Part I: Talos on Hetzner https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talo... Part II: Cilium CNI & Firewalls https://datavirke.dk/posts/…
I’d be interested to read about how you might configure cluster auto scaling with bare metal machines. I noticed that the IP address of each node are kinda hard-coded into firewall and network policy rules, so that would have to be automated somehow. Similarly with automatically spawning a load-balancer from declaring a k8s Service. I realise these things are very cloud provider specific but would be interested to see if any folks are doing this with bare metal. For me, the ease of autoscaling is one of the primary benefits of k8s for my specific workload.
I also just read about Sidero Omni [1] from the makers of Talos which looks like a Saas to install Talos/Kubernetes across any kind of hardware sourced from pretty much any provider — cloud VM, bare metal etc. Perhaps it could make the initial bootstrap phase and future upgrades to these parts a little easier?
[1]: https://www.siderolabs.com/platform/saas-for-kubernetes/
Earlier quoted context omitted.
"It's not a mess! it was designed by committee!" I'm not sure what to say here. The kubernetes docs and code speak for themselves. If you actually think that it's clean, simple, well designed, and easy to operate, with smooth interop between the parts, I can't change your mind. But in practice, I have found it very unpleasant. It seems this is common, and the usual suggestion is to pay someone else to operate it.
First you were complaining that it was ad hoc and informal. Now you seem to be complaining that it's too formal and designed by committee. Also I never said Kubernetes was well-designed, easy, or simple.
I recently rebuilt my Kubernetes cluster running across three dedicated servers hosted by Hetzner and decided to document the process. It turned into a (so far) 8-part series covering everything from bootstrapping and firewalls to setting up persistent storage with Ceph. Part I: Talos on Hetzner https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talo... Part II: Cilium CNI & Firewalls https://datavirke.dk/posts/…
Great write up and what I especially enjoyed was how you kept the bits where you ran into the classic sort of issues, diagnosed them and fixed them. The flow felt very familiar to whenever I do anything dev-opsy. I’d be interested to read about how you might configure cluster auto scaling with bare metal machines. I noticed that the IP address of each node are kinda hard-coded into firewall and network policy rules,…
I haven't used Sidero Omni yet, but if it's as well architected as Talos is, I'm sure it's an excellent solution. It still leaves open the question of ordering and provisioning the servers themselves. For simpler use-cases it wouldn't be too difficult to hack together a script to interact with the Hetzner Robot API to achieve this goal, but if I wanted any level of robustness, and if you'll excuse the shameless plug, I think I'd write a custom operator in Rust using my hrobot-rs[2] library :)
As far as the hard-coded IP addresses goes, I think I would simply move that one rule into a separate ClusterWideNetworkPolicy which is created per-node during onboarding and deleted again after. The hard-coded IP addresses are only used before the node is joined to the cluster, so technically the rule becomes obsoleted by the generic "remote-node" one immediately after joining the cluster.[3]
[1] https://github.com/hetznercloud/hcloud-cloud-controller-mana...
[2] https://github.com/MathiasPius/hrobot-rs
[3] https://github.com/MathiasPius/kronform/blob/main/manifests/...
Earlier quoted context omitted.
I used to do something similar at a previous company and this works well if you don't have to worry about scaling. YAGNI principal and all that. When you run hundreds of containers for different workloads, k8s bin packing and autoscaling (both on the pod and node level) tips the balance in my experience.
Yeah if we ever need to autoscale then I can see Kubernetes being useful, but I'd be surprised if this a problem most companies face. Even when working at StackOverflow (serving 1B+ pages, 55TB /mo [1]) did we need any autoscaling solution, it ran great on a handful of fixed servers. Although they were fairly beefy bare metal servers which I'd suspect would require significantly more VMs if it was to run on the Cloud…
GKE is IMO the best k8s solution PAAS wise that exists, but quite frankly few companies need that much control and granularity in their infrastructure.
My entire infrastructure now is AWS ECS and it autoscales and I literally never, ever, ever have had to troubleshoot it outside of my own configuration mishaps. I NEVER get on call alerts. I'm the Staff SRE at my corp.
I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.
curious what do you mean? To me Postgresql doesn't have disadvantages over SQLite, everything is just better..