Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

K3s – Lightweight Kubernetes

k3s.io

Post reply on HN
Print view View original

161–170 of 196 posts

Re: K3s – Lightweight Kubernetes

#161

Post by illamint »

_w1tm wrote: post #40
diggan wrote: post #17

"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them. Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Although, it's hard to recommend Nomad for future uses as i…

> Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Does Nomad expose an API that you can extend with controllers running inside the cluster? Because Kubernetes without operators is not Kubernetes.

> Because Kubernetes without operators is not Kubernetes.

Alright, so, what is it, then? I've been unfortunate to work at firms that generally have a minimal level of competency with Kubernetes, but across several billion dollars worth of firms, not a one has used operators in any capacity, but they leverage Kubernetes substantially. Help me understand the gap, would you?

Re: K3s – Lightweight Kubernetes

#162

Post by sharts »

OhSoHumble wrote: post #77
diggan wrote: post #17

"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them. Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Although, it's hard to recommend Nomad for future uses as i…

I've commented on this before in a different k8s thread (one about a k8s outage) but something that bears repeating is that the entire job market is Kubernetes. My personal experience is that it is very, very hard to find a job right now if your professional experience is primarily non-k8s orchestration systems. Most job positions out there require deep Kubernetes knowledge as well as hands-on experience with differe…

If that is so then the industry has a lot to recon with soon because that would mean most at chasing a shiny new thing for absolutely all the wrong reasons.

K8s is complete overkill at best for the majority of companies/workloads and introduces lots of other dependencies on teams in terms of workflows and architectures to make things not be a dumpster fire for all but very mature teams.

Re: K3s – Lightweight Kubernetes

#163

Post by arcastroe »

iamdbtoo wrote: post #134
imiric wrote: post #124

Earlier quoted context omitted.

Thanks for your perspective. How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nig…

I can't speak to performance because the workloads aren't really intense, but I run a small 3 node cluster using k3s and Longhorn and Longhorn has been really great. It was easy to setup and has been reliably running with very minimal maintanence since.

If you use longhorn, make sure to enable the network policies when installing the helm chart. For some odd reason, these are disabled by default, which means ANY pod running on your cluster has full access to the longhorn manager, API, and all your volumes

https://github.com/longhorn/charts/blob/v1.5.x/charts/longho...

Re: K3s – Lightweight Kubernetes

#164

Post by plagiarist »

hobofan wrote: post #126
umvi wrote: post #102

Earlier quoted context omitted.

A lot of smaller SaaS companies avoid k8s like the plague and only start using it once they scale to a certain size. K8s might be ubiquitous past a certain scale, but there are many jobs operating below the threshold for which the complexity of your system with k8s outweighs the complexity of your system without k8s.

If you find yourself on the Google cloud for some reason, GKE Autopilot is such a great hands-off solution that most alternative solutions will probably be more complex. I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual…

Installing an ingress controller that will do SSL is a steep cliff preventing use with so many cloud Kubernetes offerings. I don't know why it needs to have dozens of steps, roles, and resource definitions.

Re: K3s – Lightweight Kubernetes

#165

Post by davkan »

organsnyder wrote: post #84
intangible wrote: post #78

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environ…

Funny: I've been running a Talos cluster for the past six months, and just today decided to look into k3s. Talos has a lot of really nice things, but I have found that the lack of shell access can be frustrating at times when trying to troubleshoot.

Curious, what have you run into that you couldn’t troubleshoot with a privileged pod with host networking?

Re: K3s – Lightweight Kubernetes

#166

Post by hobofan »

umvi wrote: post #145
hobofan wrote: post #126

Earlier quoted context omitted.

If you find yourself on the Google cloud for some reason, GKE Autopilot is such a great hands-off solution that most alternative solutions will probably be more complex. I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual…

I use GCP, and it's hard for me to imagine something simpler than docker containers in Cloud Run which works for most of my use cases.

Cloud Run is just Knative which is also Kubernetes, so in the end it comes out to about the same. I used to use Cloud Run, but in the end there were usually always some things that ended up making GKE the more convenient choice (e.g. access to Helm charts).

Re: K3s – Lightweight Kubernetes

#167

Post by hobofan »

plagiarist wrote: post #164
hobofan wrote: post #126

Earlier quoted context omitted.

If you find yourself on the Google cloud for some reason, GKE Autopilot is such a great hands-off solution that most alternative solutions will probably be more complex. I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual…

Installing an ingress controller that will do SSL is a steep cliff preventing use with so many cloud Kubernetes offerings. I don't know why it needs to have dozens of steps, roles, and resource definitions.

Yeah, though I have to admit that even though you get SSL certificates "out of the box" with a GKE + GCP load balancer setup, their provisioning has always been really slow for me (usually 1-2h, sometimes half a day). In comparison self-installed nginx-ingress + Let's Encrypt provisions in minutes.

Re: K3s – Lightweight Kubernetes

#168

Post by moondev »

thelastparadise wrote: post #27
hhh wrote: post #13

Earlier quoted context omitted.

K3s/k8s may help with that, but it's going to be a learning curve. I personally moved onto k8s for a similar reason, but it was a learning curve

It's a learning curve, but it is consistent, reliable, and standardized. Kubernetes has become an interface, independent of implementation. It's much like how POSIX is a standard and there are many implementations. And yes, POSIX has a learning curve too, but I'd rather learn that than anything proprietary, or non standard/rapidly changing.

No doubt. The ability to scale is great, but the best thing about Kubernetes is really the API.

Re: K3s – Lightweight Kubernetes

#169

Post by eropple »

umvi wrote: post #146
eropple wrote: post #131

Earlier quoted context omitted.

Because when your docker server gets smoked, what happens? If you're using Swarm or Nomad or k8s, there's an argument there. `docker run` isn't. I've very recently moved over to a home k3s cluster--a couple old desktops and some relatively new ARM SBCs with NVMe slots. I don't use it at work, at least not directly, but it's been the least-painful solution I've found to run stuff; once you understand the model (not tr…

> Because when your docker server gets smoked, what happens? In the context of local or home servers: my raspberry pi currently has like a 3 year uptime now, but if it randomly croaked I'd just buy a new raspberry pi, docker pull the image, and get it up and running again. So you are saying the main benefit of k8s is that if some of your hardware in the cluster dies then your services are automatically still up and r…

The main benefits for me:

- I have three control planes, not one, for about 20 x86-64 and 24 arm64 cores and ~160GB of RAM across 7 nodes. For much the same reason as my NAS uses raidz2, I don't have to drop everything and fix the universe if something fails because Home Assistant just keeps going.

- With Longhorn, I have three instances of every data volume in the cluster. Longhorn also provides incremental data backups to an S3 server--Minio on the NAS, in my case, which replicates to an offsite.

- I can run things like CrunchyData's Postgres operator, which provides a solid baseline setup and additional features like point-in-time restore--and as I run personal but public-facing apps on the cluster this is beneficial.

- Logging and monitoring are centralized and were easy to do. I don't have to go attach to a container, I just go look at Grafana.

- I have a consistent interface for working with all my applications, and I like it more than Swarm's by a lot.

By having this cluster, made out of spare computers, some cheap NVMe drives, and a couple SBCs I wanted to experiment with anyway,I've removed everything I care personally about from cloud providers except for offsite backup and my email sending. I know where it is, I can hack on projects trivially, and I know it'll be there tomorrow so long as the house doesn't burn down.

And I didn't learn any of k8s at work.

Re: K3s – Lightweight Kubernetes

#170

Post by proxysna »

themgt wrote: post #85
diggan wrote: post #17

"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them. Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Although, it's hard to recommend Nomad for future uses as i…

Nomad seems to be a truly "lightweight Kubernetes" k3s requirements: 1 node, 512MB RAM, 1 CPU core Nomad requirements: "Nomad servers may need to be run on large machine instances. We suggest having between 4-8+ cores, 16-32 GB+ of memory, 40-80 GB+ of fast disk and significant network bandwidth." https://docs.k3s.io/installation/requirements https://developer.hashicorp.com/nomad/docs/install/productio...

This is for large deployments in prod. Nomad and Consul servers at home are running on an old Rpi3 with 2G of ram managing several devices and about 10 vm's. Larger deployment of ~30vm's was managed with a nomad/consul cluster of 3x2cpu4ram with no issues.
Post reply on HN