Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

K3s – Lightweight Kubernetes

k3s.io

Post reply on HN
Print view View original

151–160 of 196 posts

Re: K3s – Lightweight Kubernetes

#151

Post by sgarland »

intangible wrote: post #78

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environ…

Re: Talos persistent storage, why not run it as a VM and pass in block devices from the hypervisor? You also then gain the benefit of templated VMs that you can easily recreate or scale as needed.

Re: K3s – Lightweight Kubernetes

#152

Post by sleepybrett »

umvi wrote: post #146
eropple wrote: post #131

Earlier quoted context omitted.

Because when your docker server gets smoked, what happens? If you're using Swarm or Nomad or k8s, there's an argument there. `docker run` isn't. I've very recently moved over to a home k3s cluster--a couple old desktops and some relatively new ARM SBCs with NVMe slots. I don't use it at work, at least not directly, but it's been the least-painful solution I've found to run stuff; once you understand the model (not tr…

> Because when your docker server gets smoked, what happens? In the context of local or home servers: my raspberry pi currently has like a 3 year uptime now, but if it randomly croaked I'd just buy a new raspberry pi, docker pull the image, and get it up and running again. So you are saying the main benefit of k8s is that if some of your hardware in the cluster dies then your services are automatically still up and r…

if you only have one master sure.

The benefit to me is if one of my nodes physically dies i have enough room on my cluster to still run all my pods, they just automagically reschedule to another node.

If my master dies then I have to do a little extra work, but not much.

Re: K3s – Lightweight Kubernetes

#153

Post by mhio »

praveenhm wrote: post #66

I've been closely following the discussion on k3s and Kubernetes in general. I recently acquired an M1 Mac Ultra, and I'm curious about the best options available for running Kubernetes locally on it. Does anyone have experience or recommendations in this area? I've heard about a few tools but wanted to gather insights from this knowledgeable community.

"colima" and it's underlying project "lima" are a pretty quick way to get started.

Extremely quick to stand up a single node cluster, or many types of VMs in lima.

https://github.com/lima-vm/lima

    limactl start template://k3s
https://github.com/abiosoft/colima

    colima start --kubernetes
The tools are a bit rough around the edges if you try and do something outside of the happy path with them. Nothing bad as such, just the user experience isn't as seamless when say running the VMs on custom, addressable host networks or managing vms with launchd.

Re: K3s – Lightweight Kubernetes

#154

Post by imiric »

MPSimmons wrote: post #147
imiric wrote: post #124

Earlier quoted context omitted.

Thanks for your perspective. How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nig…

I've run Rook/Ceph, and I run Longhorn right now. I wish I didn't, and I'm actively migrating to provider-managed PVs. My advice for on-prem is to buy storage from a reliable provider with a decent history of hybrid flash/ssd, so that you can take advantage of storage tiering (unless you just want to go all flash, which is a thing if you have money). If you must use some sort of in-cluster distributed storage solutio…

Good points, thanks. What makes you wish you didn't use Rook/Ceph/Longhorn?

In a professional setting, and depending on scale, I'd probably rely on a storage provider to manage this for me. But since this is for my homelab, I am interested in a DIY solution. As a learning experience, to be sure, but it should also be something that ideally won't cause maintenance headaches.

Keeping separate volumes makes sense. I can picture three tiers: SSDs outside of the distributed storage dedicated to the hosts themselves, SSDs part of distributed storage dedicated to the services running on k3s, and HDDs for the largest volume dedicated to long-term storage, i.e. the NAS part. Eventually I might start moving to SSDs for the NAS as well, but I have a bunch of HDDs currently that I want to reuse, and performance is not critical in this case.

Re: K3s – Lightweight Kubernetes

#155

Post by imiric »

sgarland wrote: post #150
imiric wrote: post #124

Earlier quoted context omitted.

Thanks for your perspective. How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nig…

Ceph is a nightmare if you don’t set it up exactly how the docs say - and in fairness, the docs are excellent. My advice, having done Ceph/Rook, Longhorn, and now Ceph via Proxmox is the latter, assuming you have access to an actual host. Proxmox-managed Ceph is a dream, and exposing it to VMs and then K8s via RBD is easy. Longhorn is fairly easy to set up, but its performance is terrible in comparison.

Thanks for the insight. I've tried Proxmox before, and as much as I appreciate what it does, it's mostly a black box. I prefer a more hands-on approach, as long as the docs are comprehensible, and the solution is simple enough to manage without a special degree. Ceph always seemed daunting, and I've mostly ruled it out, which is why these k8s-native solutions are appealing.

Good to know about the performance. This is not critical for my use case, but I guess I'll need to test it out for myself, and see whether it's acceptable.

Re: K3s – Lightweight Kubernetes

#156

Post by MPSimmons »

imiric wrote: post #154
MPSimmons wrote: post #147

Earlier quoted context omitted.

I've run Rook/Ceph, and I run Longhorn right now. I wish I didn't, and I'm actively migrating to provider-managed PVs. My advice for on-prem is to buy storage from a reliable provider with a decent history of hybrid flash/ssd, so that you can take advantage of storage tiering (unless you just want to go all flash, which is a thing if you have money). If you must use some sort of in-cluster distributed storage solutio…

Good points, thanks. What makes you wish you didn't use Rook/Ceph/Longhorn? In a professional setting, and depending on scale, I'd probably rely on a storage provider to manage this for me. But since this is for my homelab, I am interested in a DIY solution. As a learning experience, to be sure, but it should also be something that ideally won't cause maintenance headaches. Keeping separate volumes makes sense. I can…

>Good points, thanks. What makes you wish you didn't use Rook/Ceph/Longhorn?

It seems like my volumes are constantly falling into degraded and then rebuilding. Resizing volumes requires taking the workload that's attached down, and then it seems to take forever (15m+) for my clusters to figure out that the pod is gone and a new pod is trying to attach.

Really, it's a PITA and all of the providers' storage classes seem better than Longhorn. Ceph I had less experience with but very similar problems - long-gone pods held a lock on PVCs that had to be manually expunged, or wait for a very long timeout.

Re: K3s – Lightweight Kubernetes

#157

Post by vosper »

ownagefool wrote: post #90
proxysna wrote: post #54

Earlier quoted context omitted.

You can have sd and secrets with just nomad now, it is perfect for small projects in that way. K8s also can do secrets! base64 encoded secrets! But in case you need a more secure solution. Please pull in a Vault helm chart or mess around with sops or whatever. Want to scale out? See how long it will take you until you will have to move your etcd to dedicated hw/vm's. And let me tell you - setting up external etcd in…

K8s supports encryption at rest, ACLs and audit logs. What it actually lacks is versioning, and a dynamic secrets engine, though you could build that with an operator, and things like the postgres-operator do. You also don't get the whole sealing thing, but I'd argue that's more annoying than useful.

What's "sealing" in this context?

Re: K3s – Lightweight Kubernetes

#158

Post by worksonmine »

tarruda wrote: post #44
worksonmine wrote: post #32

Earlier quoted context omitted.

The amount of services running is not an indication of how lightweight the underlying tech is. K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. Are you trying to say that 10 services and 2 nodes would be less on Nomad?

> K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. That has not been my experience with k3s. Unlike normal distributions, k3s servers do allow workload to run on them unless you manually disable. From https://docs.k3s.io/datastore/ha-embedded : " An HA K3s cluster with embedded etcd is composed of: - Three or more server node…

Yes, you can ignore the warnings and change the defaults, like I said?

> unless you manually change it

I ran into this myself just the other day.

Re: K3s – Lightweight Kubernetes

#159

Post by worksonmine »

sleepybrett wrote: post #112
tarruda wrote: post #44

Earlier quoted context omitted.

> K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. That has not been my experience with k3s. Unlike normal distributions, k3s servers do allow workload to run on them unless you manually disable. From https://docs.k3s.io/datastore/ha-embedded : " An HA K3s cluster with embedded etcd is composed of: - Three or more server node…

This is not true of k8s in general, maybe certain on prem 'distributions' enforce this, i'm not sure. Sometimes kubernetes admins/operators who are running their own control planes and etcd will run etcd on non k8s controlled nodes, or they will run it on the same nodes that run their control plane services (scheduler, apiserver, controller manager, etc). Often those nodes will be tainted in such a way that they don'…

Yeah, design was probably a poor choice of words, s/design/default/.

Re: K3s – Lightweight Kubernetes

#160

Post by moondev »

proxysna wrote: post #56
_w1tm wrote: post #40

Earlier quoted context omitted.

> Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Does Nomad expose an API that you can extend with controllers running inside the cluster? Because Kubernetes without operators is not Kubernetes.

yes, you can do that. Here's an example. https://andydote.co.uk/2021/11/22/nomad-operator-pattern/

> While Nomad doesn’t support the idea of Custom Resource Definitions, we can achieve an operator by utilising a regular Nomad job and the nomad HTTP API.

Kubernetes operator is a combination of a CRD reconciled by a controller. Without the CRD it's simply a controller.

I do like the linked post, it's well written and a valiant effort, but not close to comparable.

Post reply on HN