141–150 of 196 posts
I had a great k3s experience with a silly, over engineered weekend project to automate my fog machine with motion sensors. I connected motion sensors to battery & wifi enabled RPis, built a remote circuit to control the fog machine, and ran a k3s cluster with NATS to bring it all together. 10/10 would do again. https://blog.apartment304.com/fog-machine-madness/
Earlier quoted context omitted.
A lot of smaller SaaS companies avoid k8s like the plague and only start using it once they scale to a certain size. K8s might be ubiquitous past a certain scale, but there are many jobs operating below the threshold for which the complexity of your system with k8s outweighs the complexity of your system without k8s.
If you find yourself on the Google cloud for some reason, GKE Autopilot is such a great hands-off solution that most alternative solutions will probably be more complex. I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual…
And if you want to use EKS-A (EKS-Anywhere) it's even more effort to do the AWS IRSA setup.
It also has Traefik set up with sane defaults, and the local path provisioner is also pretty good, too. But recently I've moved to Longhorn since I plan to eventually scale past 1 node. My only complaint about Longhorn is that applications that are write-heavy and delete old data (e.g. Prometheus with short retention) will require aggressive trimming (e.g. trim once a day) to keep the actual size of volumes down. Besides that, Longhorn makes backups to S3-compatible storage very effortless and you get RWX volumes, too!
Regarding k3s itself, you can persist modifications to the way some components (e.g. Traefik) are installed through a HelmChartConfig CRD. This is what I personally use so I can use Traefik to route SSH traffic for Forgejo. Another nice thing is that although components like kube-proxy are baked in to the single k3s binary, you can still scrape metrics with Prometheus provided that you expose their endpoints somewhere on your cluster network.
Kubernetes seems like an "acquired taste". Either you use it a lot (likely at work) and are therefore comfortable using it outside of work as your one-size-fits-all solution to everything, or else you don't use it a lot and see it as a complexity behemoth that hides all of your system behind esoteric CLIs and config files that just make everything a nightmare to manage.
I'm tired of seeing comments with an apparent layer of objectivity which very quickly goes away as you reason about it for a second. You could've just said "I don't like K8s, it's too complex for my taste" but then your comment would be gray for being short and concise, so you decide to say this. Opinions are fine and you may as well hate K8s or any tech being discussed. I just hope to see less of "people who like th…
I'm trying to grapple with the fact that k8s is apparently ultra popular technology, yet every time I try to read up on it or start using it, it quickly turns into a suck fest in terms of learning curve and complexity. So my conclusion is that the people using it "for fun" must have acquired a taste for it first by using it all day at work and I wanted to see if anyone who acquired the taste might chime in as to what made them see the light.
Earlier quoted context omitted.
A lot of smaller SaaS companies avoid k8s like the plague and only start using it once they scale to a certain size. K8s might be ubiquitous past a certain scale, but there are many jobs operating below the threshold for which the complexity of your system with k8s outweighs the complexity of your system without k8s.
If you find yourself on the Google cloud for some reason, GKE Autopilot is such a great hands-off solution that most alternative solutions will probably be more complex. I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual…
Earlier quoted context omitted.
Right, you've acquired the taste. But why should people who haven't acquired the taste do this? To me seems like a mountain of unneeded complexity compared to just running a local webserver directly in a docker container with "docker run --restart always ".
Because when your docker server gets smoked, what happens? If you're using Swarm or Nomad or k8s, there's an argument there. `docker run` isn't. I've very recently moved over to a home k3s cluster--a couple old desktops and some relatively new ARM SBCs with NVMe slots. I don't use it at work, at least not directly, but it's been the least-painful solution I've found to run stuff; once you understand the model (not tr…
In the context of local or home servers: my raspberry pi currently has like a 3 year uptime now, but if it randomly croaked I'd just buy a new raspberry pi, docker pull the image, and get it up and running again.
So you are saying the main benefit of k8s is that if some of your hardware in the cluster dies then your services are automatically still up and running? Isn't a home k8s setup vulnerable to the same failure as well if the master node gets smoked?
Seems like a lot of complexity cost for something that happens extremely rarely in a home setup at least (I could see how it would be useful for high reliability cloud systems).
I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environ…
Thanks for your perspective. How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nig…
My advice for on-prem is to buy storage from a reliable provider with a decent history of hybrid flash/ssd, so that you can take advantage of storage tiering (unless you just want to go all flash, which is a thing if you have money).
If you must use some sort of in-cluster distributed storage solution, I would advise you to exclude members of your control plane from taking part, and I would also dedicate entirely separate drives and volumes for the storage distribution so that normal host workload doesn't impact latency and contention for the distributed storage.
I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environ…
To me it seems strange that a systemd unit is used, but I didn't know if I was missing something about the way MicroOS worked.
[0]: https://en.opensuse.org/SDB:K3s_cluster_deployment_on_MicroO...
I had a great k3s experience with a silly, over engineered weekend project to automate my fog machine with motion sensors. I connected motion sensors to battery & wifi enabled RPis, built a remote circuit to control the fog machine, and ran a k3s cluster with NATS to bring it all together. 10/10 would do again. https://blog.apartment304.com/fog-machine-madness/
No pictures of fog machine and spooky graveyard -- 4/10
I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environ…
Thanks for your perspective. How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nig…
My advice, having done Ceph/Rook, Longhorn, and now Ceph via Proxmox is the latter, assuming you have access to an actual host. Proxmox-managed Ceph is a dream, and exposing it to VMs and then K8s via RBD is easy.
Longhorn is fairly easy to set up, but its performance is terrible in comparison.