Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

K3sup – bootstrap K3s over SSH in < 60s

github.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: K3sup – bootstrap K3s over SSH in < 60s

#21

Post by KronisLV »

8n4vidtmkvmk wrote: post #16

I'm trying to understand why people are spinning up so many k8s clusters that they need a tool to do it for them? I have one. And it's managed. I don't think there's significant cost savings to going unmanaged, but maybe. Even so, why would I need a ton of them?

> And it's managed.

Can’t use cloud stuff on-prem and also if your clients have a server room of their own. Same for homelab.

Also it’s nice not to shift the pets attitude from servers to clusters and instead treat everything as cattle - provided you have backups of persistent data and the config versioned in a Git repo and there’s maybe some Ansible in the mix, being able to recreate an environment in the case of a fuckup is nice and also helps against bit rot.

Disclaimer: I actually prefer Docker Swarm/Compose over K8s due to simplicity (which matches my deployments and scale), but in the cases where I had to use a variety of K8s, going for K3s was pretty okay.

Re: K3sup – bootstrap K3s over SSH in < 60s

#22

Post by debarshri »

ay wrote: post #20
debarshri wrote: post #5

You can pretty install it without ssh under 60s. The fun starts after it has been installed. We have been running into lot of issues at production with k3s. There I embarked on journey to writing a kubernetes compliant and equivalent platform in rust with the help of claude [1]. It is a fun little project for now, still figuring out stuff, idea is to keep it minimal and single binary every embedded including CNI, and…

Very interesting! Architecturally - where do you run Postgres ? I assume it would be external to the cluster ? (doing it internally would create a circular dependency ?)

Yes, it is external to the cluster.

If you want to do a quick setup, it creates a SQLite DB for the metadata.

Re: K3sup – bootstrap K3s over SSH in < 60s

#23

Post by debarshri »

KronisLV wrote: post #21
8n4vidtmkvmk wrote: post #16

I'm trying to understand why people are spinning up so many k8s clusters that they need a tool to do it for them? I have one. And it's managed. I don't think there's significant cost savings to going unmanaged, but maybe. Even so, why would I need a ton of them?

> And it's managed. Can’t use cloud stuff on-prem and also if your clients have a server room of their own. Same for homelab. Also it’s nice not to shift the pets attitude from servers to clusters and instead treat everything as cattle - provided you have backups of persistent data and the config versioned in a Git repo and there’s maybe some Ansible in the mix, being able to recreate an environment in the case of a…

If you peel off all the layers in Docker Swarm and K8s, technically it has the same level of complexity. In k8s there are a lot of concepts. I would argue you have the same network, storage, and compute complexities as an operator.

Re: K3sup – bootstrap K3s over SSH in < 60s

#24

Post by [deleted] »

ay wrote: post #20
debarshri wrote: post #5

You can pretty install it without ssh under 60s. The fun starts after it has been installed. We have been running into lot of issues at production with k3s. There I embarked on journey to writing a kubernetes compliant and equivalent platform in rust with the help of claude [1]. It is a fun little project for now, still figuring out stuff, idea is to keep it minimal and single binary every embedded including CNI, and…

Very interesting! Architecturally - where do you run Postgres ? I assume it would be external to the cluster ? (doing it internally would create a circular dependency ?)

[deleted]

Re: K3sup – bootstrap K3s over SSH in < 60s

#25

Post by finaard »

debarshri wrote: post #8
pinkgolem wrote: post #6

Earlier quoted context omitted.

Do you have a writeup what problems you ran into?

We do, let me check with my team and post it here. There were many issues. On top of my mind was, after a DR drill where in a VM was booted, node did not join the cluster. Apart from that bunch of issues due to etcd, longhorn. Another major one was the CNI stopped work for a particular node. Garbage collection for images was another, we labelled the images, it would still remove then from the node. Bunch of these kin…

I've found a lot of issues come through somewhat naive networking setup - which is encouraged by the "just yolo it" installation instruction in the documentation. If you want to start understanding what's going on you'll end up in very weird corners very quickly. Also, if you don't want the API endpoints available to the world the documentation is not much help.

I've found things more stable if you can give a dedicated interface just for internal k3s communication. It can be a bridge interface on top of a vlan interface - but not the vlan interface itself, or some things will break in very interesting ways. Also, even when using IPv6, just stick with internal IPs and nat everything - touching internal IP ranges is no fun. Plus, if there's a chance you'd ever want to use dual stack, set it up with internal v6 addresses, and just don't use the v6 addresses for now. There's also a lot of unintuitive behaviour around dual stack networking - and lots of areas where documentation is just plain wrong.

I'm scripting our stuff with ansible - one of the more useful things was the realisation that in some areas changes which shouldn't break anything can lead to cluster communication being interrupted, which is a very interesting thing to deal with, especially when you can't pin it to that change that didn't touch anything close to that, and therefore should not be responsible. I've learned, and sprinkled checks to make sure all members can still reach each other in there now, so that at least when I break it on changes I directly know why.

Re: K3sup – bootstrap K3s over SSH in < 60s

#26

Post by nonameiguess »

debarshri wrote: post #5

You can pretty install it without ssh under 60s. The fun starts after it has been installed. We have been running into lot of issues at production with k3s. There I embarked on journey to writing a kubernetes compliant and equivalent platform in rust with the help of claude [1]. It is a fun little project for now, still figuring out stuff, idea is to keep it minimal and single binary every embedded including CNI, and…

You have to be careful trying to do this kind of thing. The problems you describe having below are problems with peripheral components, not k3s itself. The runtime handles garbage collection and image pinning. Your embedded runtime is using libcontainer, the same thing containerd uses, so the behavior should be identical. Since you support other runtimes, how they handle image pinning, if they support it at all, will vary. Whether or not you embed the CNI plugins and networking controllers, you're seemingly still using CNI since that's how container runtimes attach containers to a network, so whatever problems you had with CNI before would still happen. The DR VM not wanting to join sounds like it was probably due to etcd storing node IPs in the cluster member metadata. If you transfer that to a new host and it doesn't have the same IP, you need to first correct that metadata out of band, which no Kubernetes distro I'm aware of handles automatically but it's a simple etcdctl one-liner. You also need to make sure the client certificate you're using to authenticate with etcd is reissued with the new host IP in its IP SANs, which k3s does do automatically. If you're not using etcd, well, good in a way because it has a lot of cruft and I'm not a fan, but that will be difficult to support because the entire Kubernetes API and many third-party controllers are all designed around how etcd works. k3s doesn't actually require etcd and can use any SQL-based RDBMS thanks to its kine compatibility shim.

With all respect, "building it because I want to" and "working toward making (it) production grade" doesn't inspire a ton of confidence. k3s has been part of the CNCF for many years and its developer Darren Shepherd was the founding CTO for both cloud.com and Rancher Labs, which were acquired by Citrix and SUSE. It looks like you're running your own B2B company and hoping to swap out k3s as the underlying engine for multitenancy. That's very risky. Surely Claude can help you understand and use k3s just as readily as help you write a replacement, and I'm sure SUSE sells professional services. I have no clue what they charge but typically you're talking like $300 an hour and you'd probably only need 40 hours.

Re: K3sup – bootstrap K3s over SSH in < 60s

#28

Post by threecheese »

8n4vidtmkvmk wrote: post #16

I'm trying to understand why people are spinning up so many k8s clusters that they need a tool to do it for them? I have one. And it's managed. I don't think there's significant cost savings to going unmanaged, but maybe. Even so, why would I need a ton of them?

Because they are selling a “pro” version as part of their commercial product SlicerVM. It has more features for operating a k3s cluster.
Post reply on HN