Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Nomad for a Home Server

mrkaran.dev

Post reply on HN
Print view View original

21–30 of 153 posts

Re: Running Nomad for a Home Server

#21

Post by marvinblum »

I did/do run both myself, Kubernetes and Nomad, and it was a million times easier to set up Nomad (including Consul) on bare metal than it was to set up Kubernetes. Kubernetes offers more features, but you most likely don't need them and the increase in complexity makes it a pain to maintain. I'm running a three-node cluster on Hetzner [0] for Pirsch [1] right now and haven't had any difficulties whatsoever when upgrading and modifing the cluster. Adding Vault is a bit tricky, but I'm sure I'll get that done too. I highly recommend trying it if you think about maintaining your own infrastructure!

Let me know if you have any questions. I plan to write an article/kind-of-tutorial about the setup.

[0] https://www.hetzner.com/de/cloud

[1] https://pirsch.io/

Re: Running Nomad for a Home Server

#22

Post by mrweasel »

Maybe it's because Nomad is a commercial product as well, but it's really sad that it doesn't get more usage. Kubernetes is the new hotness, to the point where people don't care that the don't really need Kubernetes. They just know that the big boys are using it, so regardless of scale, so should they.

Kubernetes is exceedingly complex, and it needs to be, to support all it's use cases. However, I would claim that the vast majority of small project running on Kubernetes is just after one thing: Easier deployments.

These users would be just as well served, if not better by Nomad. It eliminates much of the complexity of running a Kubernetes cluster, but it still utilizes hardware better and you get a convenient way to deploy your code.

Re: Running Nomad for a Home Server

#23

Post by ascendantlogic »

parhamn wrote: post #9

Hashicorp has very cool stuff but I am not a fan of the config language they use on all their projects. It’s fine when I’m in an infrastructure type role and recency recall is fine but when Hashicorps tools are in my periphery it is a pain. Anyone share this? I guess the alternative is templated tomls/yamls/pseudo-jsons. Wish we’d all agree to one templated configuration format.

> Wish we’d all agree to one templated configuration format.

I'm going to post the obligatory XKCD comic because your comment is exactly what this was created for: https://xkcd.com/927/

Re: Running Nomad for a Home Server

#24

Post by linkdd »

jsiepkes wrote: post #8
outime wrote: post #3

The biggest problem with these posts is that authors seem to fail to see how K8s isn't targeting home servers or small companies. Sure, there's K3s which allows you to have a more minimalistic experience but it's still K8s. K8s is enterprise stuff and should be seen as such. It's complex because the problems that are being attempted to solve are complex and varied. That doesn't discount that Nomad is a great piece of…

> The biggest problem with these posts is that authors seem to fail to see how K8s isn't targeting home servers or small companies. Who would you say are in the target audience of Kubernetes? I doubt most medium to large companies I see implementing Kubernetes could be considered a good fit for Kubernetes. If you want to run on-prem / colo you are probably better of with something simpler like Nomad. If you want Kube…

> If you want Kubernetes it's probably a better idea to use a hosted Kubernetes solution like Google's offering.

This may not be true in the future with distributions like k0s[1]

[1]: https://k0sproject.io/

Re: Running Nomad for a Home Server

#25

Post by marvinblum »

humblepie wrote: post #19
trabant00 wrote: post #4

Nomad looks useful to me for use cases that have nothing to do with "a simple k8s" but more with distributed/HA systemd and cron. Just deployed a small 3 node cluster in prod last week for this: run some binaries with some parameters, make sure they restart on failure, have them move to another node if one fails or is rebooted, and don't waste resources by having a classical active/passive 2 node setup that doesn't s…

How easy is setting up nomad for this? Does this need a consul cluster too?

Yes, Nomad does not include the service mesh, so you need to set up Consul too. Setting up Consul is a little harder than Nomad, but still simple, and Nomad will integrate with it without additional configuration, so it "just works" when it discovers the Consul service.

Re: Running Nomad for a Home Server

#27

Post by Cederfjard »

ascendantlogic wrote: post #23
parhamn wrote: post #9

Hashicorp has very cool stuff but I am not a fan of the config language they use on all their projects. It’s fine when I’m in an infrastructure type role and recency recall is fine but when Hashicorps tools are in my periphery it is a pain. Anyone share this? I guess the alternative is templated tomls/yamls/pseudo-jsons. Wish we’d all agree to one templated configuration format.

> Wish we’d all agree to one templated configuration format. I'm going to post the obligatory XKCD comic because your comment is exactly what this was created for: https://xkcd.com/927/

Not really, the parent wasn’t suggesting developing a new standard, only settling on one.

Re: Running Nomad for a Home Server

#28

Post by the_angry_angel »

jsiepkes wrote: post #8
outime wrote: post #3

The biggest problem with these posts is that authors seem to fail to see how K8s isn't targeting home servers or small companies. Sure, there's K3s which allows you to have a more minimalistic experience but it's still K8s. K8s is enterprise stuff and should be seen as such. It's complex because the problems that are being attempted to solve are complex and varied. That doesn't discount that Nomad is a great piece of…

> The biggest problem with these posts is that authors seem to fail to see how K8s isn't targeting home servers or small companies. Who would you say are in the target audience of Kubernetes? I doubt most medium to large companies I see implementing Kubernetes could be considered a good fit for Kubernetes. If you want to run on-prem / colo you are probably better of with something simpler like Nomad. If you want Kube…

> I doubt most medium to large companies I see implementing Kubernetes could be considered a good fit for Kubernetes. If you want to run on-prem / colo you are probably better of with something simpler like Nomad.

Our path has been Ansible -> Ansible+Docker -> Docker Swarm -> k8s. We absolutely don't need k8s, but the other options all had downsides.

1. Nomad was on our list and probably would've been better, but there were no managed Nomad solutions at the time and it was not as widely used as other solutions

2. Our time on Swarm was /ok/, but it was more and more obvious that being on the lesser walked path was a problem, and it's future made us run away from it

3. k8s gave us a nice declarative deployment mechanism

4. We can switch to a managed solution down the road with less friction

Re: Running Nomad for a Home Server

#29

Post by 3np »

We moved from k8s to Nomad at my workplace, and I'm currently running almost all my self-hosted software on a 10-node Nomad cluster (with Consul and Vault) as well. The servers for each of the three gives plenty of headroom resource-wise when run on any recentish arm64 SBC, so you can get an HA cluster for not expensive.

If you integrate properly with them (which does take quite a bit of work with the ACL policies and certs), it really starts shining. With terraform, of course.

For these core services themselves and other OS packages, I use ansible, mostly because of the huge resources in the community.

It's fun and doesn't come with all the cognitive overhead of k8s. I'm a fan and will tell everyone they should consider Nomad.

It's obviously less mature, though. One thing that has been frustrating for a while is the networking configuration - a simple thing like controling which network interface a service should bind to (internal or external?) was supposedly introduced in 0.12 but completely broken until 1.0.2 (current version is 1.0.3).

Consul Connect is really awesome conceptually to make a service mesh, but is also just coming together.

There are really only two things I miss dearly now:

1) exposing consul-connect services made by Nomad (aka ingress gateway). It seems to be theoretically doable but requiring manual configuration of Connect and Envoy. If you want to expose a service from the mesh through e.g. an http load balancer, you need to either expose it raw (losing the security benefits) or manually plumb it (no load balancer seems to play nicely with connect without a lot of work, yet)

2) Recognize that UDP is a protocol people actually use in 2021. This is a critique of the whole industry.

Re: Running Nomad for a Home Server

#30

Post by znpy »

0xbadcafebee wrote: post #17
parhamn wrote: post #9

Hashicorp has very cool stuff but I am not a fan of the config language they use on all their projects. It’s fine when I’m in an infrastructure type role and recency recall is fine but when Hashicorps tools are in my periphery it is a pain. Anyone share this? I guess the alternative is templated tomls/yamls/pseudo-jsons. Wish we’d all agree to one templated configuration format.

> Wish we’d all agree to one templated configuration format. If you mean a standard configuration file format, that has almost never happened in the entire history of computing. There are standard data formats , sure, but to standardize a configuration file, all the applications need to be limited to the functionality expressed in a single config file format. Most applications hate that, because they want to add infi…

you basically described xml.

not sure I want to go back to that.

Post reply on HN