Running Nomad for a Home Server • Hacker News Board

Re: Running Nomad for a Home Server

#51

Post by ddreier » Mon, Feb 15, 2021, 4:41 PM UTC

alias_neo wrote: post #18

parhamn wrote: post #9

Hashicorp has very cool stuff but I am not a fan of the config language they use on all their projects. It’s fine when I’m in an infrastructure type role and recency recall is fine but when Hashicorps tools are in my periphery it is a pain. Anyone share this? I guess the alternative is templated tomls/yamls/pseudo-jsons. Wish we’d all agree to one templated configuration format.

I find it particularly hard to manage Vault from the CLI. I deployed it some time ago and set up a bunch of backends, I don't recall the names of things or how I configured them and I want to backup that config to replicate it. The auto complete will give you the top level command and that's it. I haven't looked too hard because I got fed up, but i just want to dump the entire config so I can replicate it without hav…

We use Terraform to manage as much of the configuration of Vault (and Consul and Nomad) as we can. The provider is pretty good.

This makes it a lot easier to get a 1000ft view of the configuration.

Re: Running Nomad for a Home Server

#52

Post by 3np » Mon, Feb 15, 2021, 4:43 PM UTC

marvinblum wrote: post #37

3np wrote: post #29

We moved from k8s to Nomad at my workplace, and I'm currently running almost all my self-hosted software on a 10-node Nomad cluster (with Consul and Vault) as well. The servers for each of the three gives plenty of headroom resource-wise when run on any recentish arm64 SBC, so you can get an HA cluster for not expensive. If you integrate properly with them (which does take quite a bit of work with the ACL policies an…

I'm running a Traefik instance on each node, so that I can expose a service by adding a bunch of labels. The load balancer is not part of the cluster and routes the traffic to the nodes. You might want to consider that too :)

I'm doing exactly that, actually!

Two bastion hosts/lbs sharing a virtual IP (keepalived), with two Traefik instances each (private and public). I actually schedule them through Nomad (on the host network interfaces) as well - since they solved the host networking issue I mentioned above it's properly set up with service checks. Super smooth to add and change services with consul catalog, and ACME-TLS included.

Things I don't like that make me want to try envoy instead:

* It eats A LOT of CPU once number of peers ramp up - there's an issue on their GH that this seems to have been introduced in 2.1.

* UDP is completely broken. For the time being I'm doing per-job nginxes for that until I have a better solution.

* It's very brittle. Did you ever misspell the wrong thing in your label? If so you probably also wondered why half of your services all stopped working as Traefik arbitrarily let it hijack everything.

* The thing I metioned above with Consul Connect. Each service can integrate with either but not both.

It was great for months though, but I guess I grew out of it just by the time I started properly understanding how all the configuration actually works (:

Re: Running Nomad for a Home Server

#53

Post by heipei » Mon, Feb 15, 2021, 4:57 PM UTC

superkuh wrote: post #43

Why would you run Nomad for a home server? This isn't for home server use. This is for practicing skills only required in commercial environments. Skills you only have a need for because someone pays you: because it's not fun and stupid. Taking that process home and doing all that instead of just running nginx from your distro repos on the sbc OS is cargo cult insanity for 99% of cases.

I would generally agree with you when talking about things like running a k8s/k3s cluster for hosting your weblog on your Raspi. With Nomad it's different though, setup on a single server is basically a 10-line systemd service definition and then you can start submitting Nomad jobs to it that are equally easy to write. You get things like cron jobs (which is more complex with systemd), parametrised jobs with multiple parameters (not possible with systemd), restart policies (easy to mess those up with systemd) and a really polished admin UI which allows to start/stop/restart jobs and sh-ing into running jobs / containers. And all for the overhead of a single systemd service.

Re: Running Nomad for a Home Server

#54

Post by ascendantlogic » Mon, Feb 15, 2021, 5:07 PM UTC

Cederfjard wrote: post #27

ascendantlogic wrote: post #23

Earlier quoted context omitted.

> Wish we’d all agree to one templated configuration format. I'm going to post the obligatory XKCD comic because your comment is exactly what this was created for: https://xkcd.com/927/

Not really, the parent wasn’t suggesting developing a new standard, only settling on one.

if all the current ones haven't been good enough then I think its implied that a new one would be created.

Re: Running Nomad for a Home Server

#55

Post by cbushko » Mon, Feb 15, 2021, 5:08 PM UTC

> (Not joking) You are tired of running Helm charts or writing large YAML manifests. The config syntax for Nomad jobs is human friendly and easy to grasp.

I write all of my kubernetes resources in terraform because I don't want to fight with helm charts. I was going to have to write something to monitor my deployments anyway and alert my co-workers that their deploys failed so why not just use terraform that tells you:

- what will change on deploy - fails when a deploy fails - times out

I didn't want to tell developers to kubectl apply and then watch their pods to make sure everything deployed ok when terraform does this out of the box..

Re: Running Nomad for a Home Server

#56

Post by nikisweeting » Mon, Feb 15, 2021, 5:09 PM UTC

I really liked Nomad last time I tried it (~2 years ago) but I was driven away by the subtle inconsistencies between Nomand, Consul, and Vault's HCL configs. Things that should be the same in all three (e.g. bind port, simple permissions options, etc. standard boilerplate) inexplicably had different config keys or HCL structures between them.

Maybe this has improved these days, I'll have to give it another shot.

Re: Running Nomad for a Home Server

#57

Post by [deleted] » Mon, Feb 15, 2021, 5:10 PM UTC

znpy wrote: post #30

0xbadcafebee wrote: post #17

Earlier quoted context omitted.

> Wish we’d all agree to one templated configuration format. If you mean a standard configuration file format, that has almost never happened in the entire history of computing. There are standard data formats , sure, but to standardize a configuration file, all the applications need to be limited to the functionality expressed in a single config file format. Most applications hate that, because they want to add infi…

you basically described xml. not sure I want to go back to that.

[deleted]

Re: Running Nomad for a Home Server

#58

Post by sneak » Mon, Feb 15, 2021, 5:11 PM UTC

0xbadcafebee wrote: post #17

parhamn wrote: post #9

Hashicorp has very cool stuff but I am not a fan of the config language they use on all their projects. It’s fine when I’m in an infrastructure type role and recency recall is fine but when Hashicorps tools are in my periphery it is a pain. Anyone share this? I guess the alternative is templated tomls/yamls/pseudo-jsons. Wish we’d all agree to one templated configuration format.

> Wish we’d all agree to one templated configuration format. If you mean a standard configuration file format, that has almost never happened in the entire history of computing. There are standard data formats , sure, but to standardize a configuration file, all the applications need to be limited to the functionality expressed in a single config file format. Most applications hate that, because they want to add infi…

There was pretty broad ad hoc standardization in dos/windowsland on the ini-file format, which was broadly formalized as toml years later.

Re: Running Nomad for a Home Server

#59

Post by tetha » Mon, Feb 15, 2021, 5:14 PM UTC

We're pulling the trigger tomorrow to migrate the first productive system to nomad, and launch 1-2 new products on nomad first. It's quite exciting.

We chose nomad there, because it's a business requirement to be able to self-host from an empty building due to the data we process - that's scary with K8. And K8 is essentially too much for the company. It's like 8 steps into the future from where most teams in product development and operations are. Some teams haven't even arrived at the problems ansible solves, disregard K8.

The hashicorp stack with Consul/Nomad/Vault/Terraform allows us to split this into 4-8 smaller changes and iterations, which allows everyone to adjust and stay on board (though each change is still massive for some teams). This results in buy-in even from the most skeptical operations team, who are now rolling out vault, because it's secure and solves their problems.

Something that overall really impressed me: One of our development teams has a PoC to use nomad to schedule and run a 20 year old application with windows domain requirements and C++ bindings to the COM API. Sure, it's not pretty, it's not ready to fail over, nomad mostly copies the binaries and starts the system on a prepared, domain joined windows host... but still, that's impressive. And it brings down minor update times from days to weeks down to minutes.

Being able to handle that work horse on one hand, and flexibly handling container deployments for our new systems in the very same orchestration is highly impressive.

Re: Running Nomad for a Home Server

#60

Post by Cederfjard » Mon, Feb 15, 2021, 5:15 PM UTC

ascendantlogic wrote: post #54

Cederfjard wrote: post #27

Earlier quoted context omitted.

Not really, the parent wasn’t suggesting developing a new standard, only settling on one.

if all the current ones haven't been good enough then I think its implied that a new one would be created.

Well I disagree, if nothing else for the fact that they never expressed that none was good enough, just frustration that there are too many (or more than one, really). Perhaps they’d be happy if everyone just settled on YAML or something.