how do you pronounce it? kuberthreeties?
K3s – Lightweight Kubernetes
181–190 of 196 posts
Re: K3s – Lightweight Kubernetes
#182Earlier quoted context omitted.
What you've listed for k3s is mostly included in k0s. I wouldnt go far to say k0s isnt convenient. * A helm controller is included in k0s * Etcd is bundled and bootstrapped automatically which I perfer because I dont want the overhead of the translation that Kine does. Although Kine is available for a non-etcd datastore if that is preferred. * Upgrade controller is included (autopilot). * They have a local storage pr…
Can you please elaborate on the "kine" overhead?
Kine (https://github.com/k3s-io/kine) is a shim or an external process (when not k3s) that translates the etcd api to enable compatibility with a database or alternative data store.
Kubernetes natively talks etcd, so this translation is what enables its usage with sqlite or another database, but it incurs an overhead.
I don't have specific numbers unfortunately since it was years ago I benchmarked Kine against etcd. But I had a better results with etcd both in cluster and single node.
I happened to stumble upon this paper that echos my experience. https://programming-group.com/assets/pdf/papers/2023_Lightwe... Particularly, the high controller cpu usage (even for an empty cluster), and higher latencies.
Re: K3s – Lightweight Kubernetes
#183Earlier quoted context omitted.
I've commented on this before in a different k8s thread (one about a k8s outage) but something that bears repeating is that the entire job market is Kubernetes. My personal experience is that it is very, very hard to find a job right now if your professional experience is primarily non-k8s orchestration systems. Most job positions out there require deep Kubernetes knowledge as well as hands-on experience with differe…
If that is so then the industry has a lot to recon with soon because that would mean most at chasing a shiny new thing for absolutely all the wrong reasons. K8s is complete overkill at best for the majority of companies/workloads and introduces lots of other dependencies on teams in terms of workflows and architectures to make things not be a dumpster fire for all but very mature teams.
I keep hearing this viewpoint, but I really don’t understand it. I use k3s for all sorts of personal projects, specifically because it makes things easier for me. I don’t have to worry about the underlying OS. I can focus on the application that’s actually providing value for me.
On the professional side, k8s handles orchestration issues far better than anything else I’ve worked with. Using autoscaling instances is a nightmare by comparison, and requires most of the same initial effort to do it right.
There is probably a middle ground where it requires a certain amount of complexity in the K8s configuration that isn’t worth it compared to other platforms (especially on bare metal), but I haven’t found it, yet.
I think it’s easy to forget how much goes into running even a simple application with high uptime requirements. Kelsey Hightower makes some great points about this here:
Re: K3s – Lightweight Kubernetes
#184"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them. Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Although, it's hard to recommend Nomad for future uses as i…
If Nomad provides workload scheduling, a runtime, workload networking, service networking, and some sort of overall orchestrator/controller plus an API, then it has these same services. Running them in one process instead of ten doesn't make it more lightweight.
You're free to run either on one node, but other than for toy purposes like demos and learning, why use a cluster compute system when you're not actually running a cluster?
Re: K3s – Lightweight Kubernetes
#185Earlier quoted context omitted.
Thanks for the insight. I've tried Proxmox before, and as much as I appreciate what it does, it's mostly a black box. I prefer a more hands-on approach, as long as the docs are comprehensible, and the solution is simple enough to manage without a special degree. Ceph always seemed daunting, and I've mostly ruled it out, which is why these k8s-native solutions are appealing. Good to know about the performance. This is…
> I prefer a more hands-on approach ... which is why these k8s-native solutions are appealing "K8s-native" here implies Rook, which is in no way hands-on for Ceph. > as long as ... the solution is simple enough to manage without a special degree Ceph is not simple, that's my point. I assume you've read the docs [0] already; if not, please do so _in their entirety_ before you use it. There are so many ways to go wrong…
By "k8s-native" I meant solutions that were built from the ground up with k8s in mind. Rook is built on Ceph, and as such requires knowledge and maintenance of both, which seems like much more difficult than managing something like Longhorn.
I think you misunderstood me. I'm not inclined to give Ceph/Rook a try because of its complexity, regardless of the state of its documentation. I was refering to Proxmox in my previous comment, in the sense that I don't need a VM/container manager/orchestrator with a pretty UI. If I'm already running k3s, I can manage the infrastructure via the CLI and IaaC, and removing that one layer of abstraction that Proxmox provides is a positive to me. Which is why adding just a storage backend like Longhorn to a k3s cluster seems like the path of least resistance for my use case.
Ultimately, I don't _want_ to deal with the low-level storage details. If I did, I'd probably be managing RAID, ZFS, or even Ceph. For my current NAS I just use a single JBOD server with SnapRAID and MergerFS. This works great for my use case, but I want to have pseudo-HA and better fault tolerance, and experiment with k8s/k3s in the process.
So I'm looking for a k8s-native tool that I can throw a bunch of nodes and disks at, easily configure it to serve a few volumes, and that it gives me somewhat performant, reliable, and hopefully maintenance-free, block or object-level access. Persistent storage has always been a headache in k8s land, but I'm hoping that nowadays such user friendly and capable solutions exist that will allow me to level up my homelab.
Re: K3s – Lightweight Kubernetes
#186Earlier quoted context omitted.
Can you please elaborate on the "kine" overhead?
Kine ( https://github.com/k3s-io/kine ) is a shim or an external process (when not k3s) that translates the etcd api to enable compatibility with a database or alternative data store. Kubernetes natively talks etcd, so this translation is what enables its usage with sqlite or another database, but it incurs an overhead. I don't have specific numbers unfortunately since it was years ago I benchmarked Kine against etcd…
thanks!
my problem with etcd was very high and constant I/O and CPU usage. I don't mind the latency.
Re: K3s – Lightweight Kubernetes
#187Oh yeah I always wanted to ask someone who's in the deep with kubernetes.
How do you migrate services and storage from one cluster to another?
Re: K3s – Lightweight Kubernetes
#188Earlier quoted context omitted.
There's not any common reason I can think of to go ipv6 on the backend. KISS --if there's enough IPs in a private v4 range, use them. If you need/want to serve v6 clients, put a v6 load balancer in front. You're likely going to need that lb anyway.
I think the use case was running on a cheap server that actually doesn't have ipv4 connectivity
correct
Re: K3s – Lightweight Kubernetes
#189forget about it if you want to run it on an IPv6-only VPS, it's not ready yet
K3s absolutely supports ipv6-only
this tutorial doesn't work on an IPv6-only VPS
https://www.jeffgeerling.com/blog/2022/quick-hello-world-htt...
Re: K3s – Lightweight Kubernetes
#190Earlier quoted context omitted.
> I prefer a more hands-on approach ... which is why these k8s-native solutions are appealing "K8s-native" here implies Rook, which is in no way hands-on for Ceph. > as long as ... the solution is simple enough to manage without a special degree Ceph is not simple, that's my point. I assume you've read the docs [0] already; if not, please do so _in their entirety_ before you use it. There are so many ways to go wrong…
By "k8s-native" I meant solutions that were built from the ground up with k8s in mind. Rook is built on Ceph, and as such requires knowledge and maintenance of both, which seems like much more difficult than managing something like Longhorn. I think you misunderstood me. I'm not inclined to give Ceph/Rook a try because of its complexity, regardless of the state of its documentation. I was refering to Proxmox in my pr…
Ahhh, yes, I misunderstood - sorry!
Also, I feel obligated to ask “are you me?” at your stated homelab journey. I also went from mergerfs + SnapRAID, albeit shifting to ZFS. I also went with k3s, but opted for k3os which is now dead and thus leaving me needing to shift again (I’m moving to Talos). Finally, everything in my homelab is also in IaC, with VMs buiit by Packer + Ansible, and deployed with Terraform.
Happy to discuss this more at length if you have any questions; my email is in my profile.