Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bare-Metal Kubernetes with K3s

blog.alexellis.io

Post reply on HN
Print view View original

41–50 of 136 posts

Re: Bare-Metal Kubernetes with K3s

#41

Post by sandGorgon »

polskibus wrote: post #27
sandGorgon wrote: post #20

Earlier quoted context omitted.

we are running k3s on metal in production. works great actually. we use haproxy as ingress and lb.

How stable do you see it ? What's your cluster size? How long has it been running? Any tips for how to approach starting such setup?

we scale up to about 100 machines. We use spot instances EXTENSIVELY. And that configuration was tricky actually. Its been a couple of months now. Works pretty ok.

k3s is actually pretty simple to use now. the tricky part was to integrate with https://github.com/kubernetes/cloud-provider-aws and https://github.com/DirectXMan12/k8s-prometheus-adapter

The hardest part is to get it to work with spot instances. we use https://github.com/AutoSpotting/AutoSpotting to integrate with it.

Re: Bare-Metal Kubernetes with K3s

#42

Post by xaduha »

cassianoleal wrote: post #29
xaduha wrote: post #18

Earlier quoted context omitted.

Point here is that term bare-metal cannot apply to applications relying on OS features. Unikernels are the closest thing you can use this term in this area.

Wikipedia [0] doesn't agree with you (neither do I, but who am I anyway?): "A bare-metal server is a computer server that hosts one tenant, or consumer, only.[1] The term is used for distinguishing between servers that can host multiple tenants and which utilize virtualisation and cloud hosting.[2] Such servers are used by a single consumer and are not shared between consumers. Each server may run any amount of work…

I wouldn't put much stock in page made just in 2016 that started like this https://en.wikipedia.org/w/index.php?title=Bare-metal_server...

Re: Bare-Metal Kubernetes with K3s

#44

Post by thebsdbox »

tasqa wrote: post #5

What I'd like to see for change is actually doing the bare metal part itself. I've seen so many k8s showcase posts of this or that, but never actually someone who's running it on actual servers they own and without using any big four cloud API's (I consider Equinix to be part of those too soon...) to handle the LB/Ingress/Network virtualization stuff they provide and still say it is easy to use..

The HA control plane and service type LB is all handled by https://kube-vip.io and is designed to be as transparent to the user as possible.

Re: Bare-Metal Kubernetes with K3s

#45

Post by ncrmro »

alexellisuk wrote: post #6
ncrmro wrote: post #2

I actually just got a NUC with 64gb ram. Alpine installed to encrypted NVMe zfs zpool. This runs KVM One of these guests is alpine with k3s (tried k3os, very limiting in a good way) that allows me to pass a host directory directly into the vm using p9 (tried nfs but lil heavy) So any storage needs get the benifit of regular snapshots, compression and sync to nas. Really wish it would be more know that k8 single node…

Maybe you could set up multiple K3s hosts and form a cluster with the VMs? Have you taken a look at inlets-operator that I mention in the post for hosting? My favourite party trick is exposing the IngressController's port 80 and 443. https://docs.inlets.dev/#/get-started/quickstart-ingresscont... There was a user who was paying 7 USD / mo per site to host 20 side-project sides.. expensive. They switched out to a comp…

That is the plan eventually so that I can do node upgrades with lil downtime

Re: Bare-Metal Kubernetes with K3s

#46

Post by ncrmro »

ClumsyPilot wrote: post #26
ncrmro wrote: post #2

I actually just got a NUC with 64gb ram. Alpine installed to encrypted NVMe zfs zpool. This runs KVM One of these guests is alpine with k3s (tried k3os, very limiting in a good way) that allows me to pass a host directory directly into the vm using p9 (tried nfs but lil heavy) So any storage needs get the benifit of regular snapshots, compression and sync to nas. Really wish it would be more know that k8 single node…

I think this is accurate, but thrre is great educational value when you can see several physical computers working in a cluster, scaling pod on CPU, etc. I used to have 3 low-power Nuc-style computers, and after playing around for a few months I did the same thing as you have - replaced them with a single beefier machine, and its a lot more practical.

Totes agree, I think I got the vms set up once a long time ago before k3s and it was some real work

Re: Bare-Metal Kubernetes with K3s

#47

Post by JamesSwift »

AntiImperialis2 wrote: post #15
im_down_w_otp wrote: post #12

I don't entirely understand what's happening here. The title and the post talk a lot about "Bare-Metal", but it also seems to indicate that everything is hosted on servers running Ubuntu. Which, if accurate, seems to be Kubernetes running on Linux, not bare-metal?

Bare-metal refers to running on hardware directly (vs. running on VMs or the cloud). I'm interested to know what you thought it was.

Im surprised you've been downvoted as much as you have. It appears bare metal has a very specific meaning in certain contexts. For what its worth, it also has a very specific meaning in cloud provider contexts, which is exactly what you've defined here.

Re: Bare-Metal Kubernetes with K3s

#48

Post by JamesSwift »

im_down_w_otp wrote: post #12

I don't entirely understand what's happening here. The title and the post talk a lot about "Bare-Metal", but it also seems to indicate that everything is hosted on servers running Ubuntu. Which, if accurate, seems to be Kubernetes running on Linux, not bare-metal?

It appears there's two camps in the audience. The camp I'm familiar with (and it appears the author as well) use bare-metal to mean running on your own computers (either in-house or colo) instead of in a VM on a cloud provider.

Re: Bare-Metal Kubernetes with K3s

#49

Post by pjmlp »

xaduha wrote: post #18
cassianoleal wrote: post #16

Earlier quoted context omitted.

Kubernetes relies on containers, which are made out of cgroups and namespaces. Those are Linux features, so there's no way around that. This is as bare-metal as it gets since it's not running on a VM.

Point here is that term bare-metal cannot apply to applications relying on OS features. Unikernels are the closest thing you can use this term in this area.

It is possible to run containers directly on top of type 1 hypervisors.

Re: Bare-Metal Kubernetes with K3s

#50

Post by skissane »

_jal wrote: post #19
AntiImperialis2 wrote: post #15

Earlier quoted context omitted.

Bare-metal refers to running on hardware directly (vs. running on VMs or the cloud). I'm interested to know what you thought it was.

In the dark ages, when running an OS in the way you're referring to 'bare metal' was the default, it meant running something without a traditional kernel underneath it. Not many things are written to do that, of course. Oracle used to offer an installation mode like this. It was generally a gimmick - you pay for a tiny bit of performance with a ton of flexibility. There are probably use cases where it makes sense, bu…

> Oracle used to offer an installation mode like this

Oracle, and BEA before them, used to offer a JVM which ran on top of a thin custom OS designed only to host the JVM, you could call it a "unikernel". Product was called JRockit Virtual Edition (JRVE), WebLogic Server Virtual Edition (WLS-VE, when used to run WebLogic), earlier BEA called it LiquidVM. The internal name for that thin custom OS was in fact "Bare Metal". Similar in concept to https://github.com/cloudius-systems/osv but completely different implementation

I think one thing which caused a problem for it, is a lot of customers want to deploy various management tools to their VMs (security auditing software, performance monitoring software, etc) and when your VM runs a custom OS that becomes very difficult or impossible. So adopting this product could lead to the pain of having to ask for exceptions to policies requiring those tools and then defending the decision to adopt it against those who use those policies to argue against it. I think this is part of why the product was discontinued.

Nowadays, Oracle offers "bare metal servers" [1] – which are just hypervisor-less servers, same as other cloud vendors do. Or similarly, "Oracle Database Appliance Bare Metal System" [2] – which just means not installing a hypervisor on your Oracle Database Appliance.

So Oracle seems to have a history of using the phrase "bare metal" in both the senses being discussed here.

[1] https://www.oracle.com/cloud/compute/bare-metal.html

[2] https://docs.oracle.com/en/engineered-systems/oracle-databas...

Post reply on HN