Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

K3s – Lightweight Kubernetes

k3s.io

Post reply on HN
Print view View original

171–180 of 196 posts

Re: K3s – Lightweight Kubernetes

#171

Post by emptysongglass »

pojzon wrote: post #139
sleepybrett wrote: post #106

Earlier quoted context omitted.

also with hashicorps new licensing regime...

That does not impact 99% of ppl.. The licence change impacts only companies like Digger or Spacelift that build their platform abusing TF oss offering.

That's a bad take and you know it. HashiCorp couldn't make money because their enterprise offering was slim on features and overpriced. Rather than compete, they're playing protect-the-castle with some aggressive plays to keep their company afloat. They're bad stewards of open source (always have been) and are now getting their just desserts.

Re: K3s – Lightweight Kubernetes

#172

Post by proxysna »

vosper wrote: post #157
ownagefool wrote: post #90

Earlier quoted context omitted.

K8s supports encryption at rest, ACLs and audit logs. What it actually lacks is versioning, and a dynamic secrets engine, though you could build that with an operator, and things like the postgres-operator do. You also don't get the whole sealing thing, but I'd argue that's more annoying than useful.

What's "sealing" in this context?

Unsealing is the process of obtaining the plaintext root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

Prior to unsealing, almost no operations are possible with Vault. For example authentication, managing the mount tables, etc. are all not possible. The only possible operations are to unseal the Vault and check the status of the seal.

https://developer.hashicorp.com/vault/docs/concepts/seal

Re: K3s – Lightweight Kubernetes

#173

Post by pas »

imiric wrote: post #155
sgarland wrote: post #150

Earlier quoted context omitted.

Ceph is a nightmare if you don’t set it up exactly how the docs say - and in fairness, the docs are excellent. My advice, having done Ceph/Rook, Longhorn, and now Ceph via Proxmox is the latter, assuming you have access to an actual host. Proxmox-managed Ceph is a dream, and exposing it to VMs and then K8s via RBD is easy. Longhorn is fairly easy to set up, but its performance is terrible in comparison.

Thanks for the insight. I've tried Proxmox before, and as much as I appreciate what it does, it's mostly a black box. I prefer a more hands-on approach, as long as the docs are comprehensible, and the solution is simple enough to manage without a special degree. Ceph always seemed daunting, and I've mostly ruled it out, which is why these k8s-native solutions are appealing. Good to know about the performance. This is…

Ceph is ... well, it's an amazing journey. Especially if you were there when it started, and watched as each release made it more capable & crazier.

From the earily days of trying to remember WTF the CRUSH and "rados" acronyms actually meant and focusing on network and storage concerns and [0] ... architectural optimizations [1] ... a RedHat acquisition ... adjusting to the NVMe boom [2] ... and then Rook, probably one of the first k8s operators trying to somehow operate this underwatere beast in a sane manner.

If you are interested in it ... set it up manually once (get the binaries, write config files, generate keyrings, start the monitors, set up OSDs, RGW and you can use FileZilla or any S3 client). For some more productionish usage there's a great all-in-one docker image https://quay.io/repository/ceph/ceph?tab=tags .

[0] oh don't forget to put the intra-cluster chatter on an internal network, or use a virtual interface and traffic shaping to have enough bandwidth for replication and restore, and what to do if PGs are not pairing and OSDs are not coming up]

[1] raw block device support, called BlueStore, which basically creates a GPT partitioned device, with ~4 partitions, stores the object map in LevelDB - and then later RocksDB

[2] SeaStore, using the Seastar framework, SPDK and DPDK, optimize everything for chunky block I/O in userspace

Re: K3s – Lightweight Kubernetes

#174

Post by pas »

nullify88 wrote: post #108
r-bar wrote: post #45

Earlier quoted context omitted.

K3S includes some extras that make it nice for working in small local clusters, but are not part of the standard k8s codebase. * Traefik daemonset as a load balancer * Helm controller that lets you apply helm manifests without the helm command line * Upgrade controller * Sqlite as the default backing store for the k8s API * Their own local storage provisioner K0S has a lot of the same goals: be light weight and self…

What you've listed for k3s is mostly included in k0s. I wouldnt go far to say k0s isnt convenient. * A helm controller is included in k0s * Etcd is bundled and bootstrapped automatically which I perfer because I dont want the overhead of the translation that Kine does. Although Kine is available for a non-etcd datastore if that is preferred. * Upgrade controller is included (autopilot). * They have a local storage pr…

Can you please elaborate on the "kine" overhead?

Re: K3s – Lightweight Kubernetes

#175

Post by organsnyder »

davkan wrote: post #165
organsnyder wrote: post #84

Earlier quoted context omitted.

Funny: I've been running a Talos cluster for the past six months, and just today decided to look into k3s. Talos has a lot of really nice things, but I have found that the lack of shell access can be frustrating at times when trying to troubleshoot.

Curious, what have you run into that you couldn’t troubleshoot with a privileged pod with host networking?

I had a situation where the etcd cluster got hosed, making it basically impossible (at least with the ways I know) to interact with the k8s API at all. So I didn't have any way to get a privileged pod running.

Re: K3s – Lightweight Kubernetes

#176

Post by organsnyder »

MPSimmons wrote: post #156
imiric wrote: post #154

Earlier quoted context omitted.

Good points, thanks. What makes you wish you didn't use Rook/Ceph/Longhorn? In a professional setting, and depending on scale, I'd probably rely on a storage provider to manage this for me. But since this is for my homelab, I am interested in a DIY solution. As a learning experience, to be sure, but it should also be something that ideally won't cause maintenance headaches. Keeping separate volumes makes sense. I can…

>Good points, thanks. What makes you wish you didn't use Rook/Ceph/Longhorn? It seems like my volumes are constantly falling into degraded and then rebuilding. Resizing volumes requires taking the workload that's attached down, and then it seems to take forever (15m+) for my clusters to figure out that the pod is gone and a new pod is trying to attach. Really, it's a PITA and all of the providers' storage classes see…

I've had similar issues with Mayastor (another in-cluster storage solution). It's under heavy development, so I've assumed the more mature options were better.

I'm working on v2 of my homelab cluster, and I'm going with plain old NFS to a file server with a ZFS pool. Yes, I will have a single node as a point of failure, but with how much pain I've had so far I think I'll be coming out ahead in terms of uptime.

Re: K3s – Lightweight Kubernetes

#177

Post by mariuz »

K3s and hetzner related

hetzner-k3s : This is a CLI tool to quickly create and manage Kubernetes clusters in Hetzner Cloud using the lightweight Kubernetes distribution k3s from Rancher. https://github.com/vitobotta/hetzner-k3s

Kubernetes on Hetzner Cloud the easiest way https://vitobotta.com/2023/01/07/kubernetes-on-hetzner-cloud...

Re: K3s – Lightweight Kubernetes

#178

Post by sgarland »

imiric wrote: post #155
sgarland wrote: post #150

Earlier quoted context omitted.

Ceph is a nightmare if you don’t set it up exactly how the docs say - and in fairness, the docs are excellent. My advice, having done Ceph/Rook, Longhorn, and now Ceph via Proxmox is the latter, assuming you have access to an actual host. Proxmox-managed Ceph is a dream, and exposing it to VMs and then K8s via RBD is easy. Longhorn is fairly easy to set up, but its performance is terrible in comparison.

Thanks for the insight. I've tried Proxmox before, and as much as I appreciate what it does, it's mostly a black box. I prefer a more hands-on approach, as long as the docs are comprehensible, and the solution is simple enough to manage without a special degree. Ceph always seemed daunting, and I've mostly ruled it out, which is why these k8s-native solutions are appealing. Good to know about the performance. This is…

> I prefer a more hands-on approach ... which is why these k8s-native solutions are appealing

"K8s-native" here implies Rook, which is in no way hands-on for Ceph.

> as long as ... the solution is simple enough to manage without a special degree

Ceph is not simple, that's my point. I assume you've read the docs [0] already; if not, please do so _in their entirety_ before you use it. There are so many ways to go wrong, from hardware (clock skew due to BIOS-level CPU power management, proper PLP on your drives...), to configuration (incorrect PG sizing, inadequate replication and/or EC settings...) and more.

I'm not trying to dissuade you from tackling this, I'm just saying it is in no way easy or simple. Statements like "k8s-native solutions" always make me cringe, because it usually means you want to use an abstraction without understanding the fundamentals.

To be clear, I have read the docs, set it up on my own, and decided I didn't want to try to manage it. I've ran a ZFS pool for a few years on Debian and shifted over to TrueNAS Scale last week; not because I was unable to deal with ZFS' complexity (knock on wood, the only [temporary] data loss I ever had was an incorrect `rm -rf`, and snapshots fixed that), but because of continual NFS issues. I may yet switch back; I don't know - I just no longer had the time to troubleshoot the data layer. Ceph makes ZFS looks like child's play in comparison.

[0]: https://docs.ceph.com/en/latest/

Re: K3s – Lightweight Kubernetes

#180

Post by sklarsa »

nunez wrote: post #76
praveenhm wrote: post #66

I've been closely following the discussion on k3s and Kubernetes in general. I recently acquired an M1 Mac Ultra, and I'm curious about the best options available for running Kubernetes locally on it. Does anyone have experience or recommendations in this area? I've heard about a few tools but wanted to gather insights from this knowledgeable community.

If you're just messing around, just use kind ( https://kind.sigs.k8s.io ) or minikube if you want VMs ( https://minikube.sigs.k8s.io ). Both work on ARM-based platforms. You can also use k3s; it's hella easy to get started with and it works great.

If you go the kind route, be sure to not use Docker for mac and instead opt for podman which is much more resource efficient. Now that I've switched over to podman, my computer doesn't sound like it's about to blast off when I'm running clusters.
Post reply on HN