Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Nomad for a Home Server

mrkaran.dev

Post reply on HN
Print view View original

101–110 of 153 posts

Re: Running Nomad for a Home Server

#101

Post by cbushko »

glynnforrest wrote: post #65

I only have praise for the HashiStack (Nomad, Consul, Vault). Folks often reach for Kubernetes as part of a larger initiative to containerize their apps, add ephemeral test environments, etc, but I really think Nomad is better suited to this. Choosing k8s is just the first step - you have to deal with what 'distribution' of k8s to use, upgrades, pages of yaml, secrets stored in plaintext by default... Once you've got…

Or you can pay the price for it and use GKE.

Of course, not for a home lab :)

Re: Running Nomad for a Home Server

#102

Post by chucky_z »

rektide wrote: post #67

A whole lot of anti- Kubernetes "you most likely don't need them and the increase in complexity makes it a pain to maintain" and "Kubernetes is exceedingly complex" in this thread & somewhat in this article. I agree that you probably don't need Kubernetes, and perhaps yeah it could be considered complex. But I think it's the right fit for most developers/doers & over time most operators too. Kubernetes is not Kuberne…

I agree with a lot of the points you make.

Have you looked into nomad, consul and vault; along with everything they provide?

Re: Running Nomad for a Home Server

#103

Post by zomglings »

ddreier wrote: post #49
zomglings wrote: post #36

From the article: > I always maintain "Day 0 is easy, Day N is the real test of your skills". Would be interesting to see how this applies to the author's use of Nomad. It's easy to shit on Kubernetes because of its complexity, but this article seems to be comparing the Nomad Day 0 experience with the Kubenrnetes Day N experience. I'm firmly of the opinion that you don't need much more than systemd (or equivalent) +…

The Nomad Day N experience is pretty good. I help maintain 4 Nomad clusters running approx 40k Allocations (like Pods) for work. We have basically no problems with Nomad itself, and requires pretty much no day to day intervention. Upgrades are pretty painless too. We've gone from 0.7.x to 0.12.x with these same clusters, and will be going to 1.x soon. Happy to try to answer specific questions.

Do you run other services (Vault, Consul, etc.) for service discovery, configuration management, etc.?

Genuinely curious about the load of managing this on the infrastructure team.

Re: Running Nomad for a Home Server

#104

Post by 64mb »

BurritoAlPastor wrote: post #66
throwaway894345 wrote: post #62

Earlier quoted context omitted.

I would be very interested in a more detailed write-up on Nomad vs Kubernetes for bare metal. I'm working through getting Kubernetes stood up, but I'm running into a dearth of features--namely you have to bring your own load balancer provider, storage provider, ingress controller, external DNS, monitoring, secret encryption, etc, etc before you can run any real world applications on top of it. I would be interested i…

You have to bring all of those same things to a Nomad deployment as well. It’s generally more lightweight than Kubernetes, so it might be easier to wire those other components in, but you do still need to do that work either way.

> it might be easier to wire those other components in

IMO the few lines of yaml to set the path/host for an Ingress definition seems cleaner to me than using consul-template to spit out some LB config (as in the post's example).

For simplicity, a few years ago I preferred Traefik + Swarm. Add a label or two and you're done. But Swarm died :/

Re: Running Nomad for a Home Server

#105

Post by tutfbhuf »

jokethrowaway wrote: post #99
marvinblum wrote: post #21

I did/do run both myself, Kubernetes and Nomad, and it was a million times easier to set up Nomad (including Consul) on bare metal than it was to set up Kubernetes. Kubernetes offers more features, but you most likely don't need them and the increase in complexity makes it a pain to maintain. I'm running a three-node cluster on Hetzner [0] for Pirsch [1] right now and haven't had any difficulties whatsoever when upgr…

I had the opposite experience (in 2 different companies). Setting up K8s was quite straightforward and docs were helpful. We ended up building a deployment UI for it though. Consul is nice and easy to use. Nomad has been a painful experience: the default UI is confusing (people accidentally killed live containers), we have some small bits and pieces that don't quite behave as we expect and have no idea how to fix the…

Yup. Setting up k8s with kubeadm on bare-metal is very straight forward and can be done within a few minutes on any linux host that is supported by kubeadm and docker + ssh access.

Re: Running Nomad for a Home Server

#106

Post by davestephens »

sofixa wrote: post #73
omneity wrote: post #46

Earlier quoted context omitted.

Yeah, when making the decision it was quite harrowing to think of maintaining a cluster in production. Nomad had very little operational complexity compared to what we imagined. We've had two main outages in months: - Server disks were filling up and we hadn't set up monitoring properly at the time (ironic for the name of our company :) ). Not Nomad's fault. - A faulty healthcheck caused all the servers of a cluster…

Jaeger is pretty great for tracing, and can integrate with Traefik/Envoy ( or whatever you use for ingress/inter-service communication). We're running Loki for the logs ( via nomad log forwared/shipper and promtail) and so far it's going great. I'll have to do a write-up about the the whole thing.

Would love to see that write-up!

Re: Running Nomad for a Home Server

#107

Post by awinter-py »

this is compelling:

> Nomad shines because it follows the UNIX philosophy of "Make each program do one thing well". To put simply, Nomad is just a workload orchestrator. It only is concerned about things like Bin Packing, scheduling decisions.

so is the stuff about saner config syntax

'k8s is bad at everything and nomad is only trying to be bad at one thing' actually makes sense of my reality

Re: Running Nomad for a Home Server

#108

Post by alexeldeib »

mr-karan wrote: post #79
OJFord wrote: post #68

> - Job: Job is a collection of different groups. Job is where the constraints for type of scheduler, update strategies and ACL is placed. > - Group: Group is a collection of different tasks. A group is always executed on the same Nomad client node. You'll want to use Groups for use-cases like a logging sidecar, reverse proxies etc. > - Task: Atomic unit of work. A task in Nomad can be running a container/binary/Java…

Author here. Hm, now that you point this out, I do get your point. However you can't run multiple different Pods in a deployment (you can run multiple containers in a Pod), that's why a Job isn't really comparable to a ReplicaSet. I could very well be wrong, but this is my understanding.

I think the tricky part is Nomad merges replica count + definition into Group, while K8s separates Pod/RS/Deployment (and uses RS to orchestrate upgrades, which I think Nomad handles on its own somehow?).

Job ~= Deployment

Group ~= Replicaset/Pod combined

Task ~= container in pod

Re: Running Nomad for a Home Server

#109

Post by ddreier »

zomglings wrote: post #103
ddreier wrote: post #49

Earlier quoted context omitted.

The Nomad Day N experience is pretty good. I help maintain 4 Nomad clusters running approx 40k Allocations (like Pods) for work. We have basically no problems with Nomad itself, and requires pretty much no day to day intervention. Upgrades are pretty painless too. We've gone from 0.7.x to 0.12.x with these same clusters, and will be going to 1.x soon. Happy to try to answer specific questions.

Do you run other services (Vault, Consul, etc.) for service discovery, configuration management, etc.? Genuinely curious about the load of managing this on the infrastructure team.

Yep, we run the full stack. Consul for service discovery and as the storage backend for Vault. We use Vault for config, PKI, Nomad/Consul ACL auth, and we're just starting to experiment with MSSQL dynamic credentials.

Of the three systems, Vault probably takes the most of our time and effort, and that's probably only a few hours per month. We've struggled a bit with performance at least partially because the Consul backend is shared with service discovery.

All of the VMs are built and managed with Terraform using images built with Packer+Ansible. We also use the Nomad/Consul/Vault Terraform providers to apply and manage their configurations.

We have an SRE/Platform Engineering team of 12 (and hiring) that's responsible for the overall orchestration platform additionally including Prometheus/Thanos/Grafana for metrics and ELK for logs.

Hope that's helpful!

Re: Running Nomad for a Home Server

#110

Post by jordanbeiber »

throwaway894345 wrote: post #62
marvinblum wrote: post #21

I did/do run both myself, Kubernetes and Nomad, and it was a million times easier to set up Nomad (including Consul) on bare metal than it was to set up Kubernetes. Kubernetes offers more features, but you most likely don't need them and the increase in complexity makes it a pain to maintain. I'm running a three-node cluster on Hetzner [0] for Pirsch [1] right now and haven't had any difficulties whatsoever when upgr…

I would be very interested in a more detailed write-up on Nomad vs Kubernetes for bare metal. I'm working through getting Kubernetes stood up, but I'm running into a dearth of features--namely you have to bring your own load balancer provider, storage provider, ingress controller, external DNS, monitoring, secret encryption, etc, etc before you can run any real world applications on top of it. I would be interested i…

It’s much easier with nomad as you are not forced in to a ”black box” with networking layers and surrounding requirements.

Bare metal nomad - use with consul and hook up traefik with consul backend. This would be the simplest, most ”zero conf”, way to go.

I’ve used this setup for a few years heavy production use (e-commerce & 50 devs)

As consul presents SRV records you can hook up a LB using those, or use nomad/consul templating to configure one.

Service mesh with mTLS is actually rather approachable and we’ve deployed it on selected services where we need to track access and have stricter security. (This however had us move off traefik and in to nginx + openresty)

Now if you want secrets management on steroids you’ll want vault. It’s really in many ways at the heart of things. It raises complexity, but the things that you can do with the nomad/consul/vault stack is fantastic.

Currently we use vault for ssl pki, secrets management for services & ci/cd, and ssh cert pki.

These things really form a coherent whole and each component is useful on it own.

Compared to k8s it’s a much more versatile stack although not as much of a “framework” and more like individual “libs”.

I always come back to the description: “more in line with the unix philosophy”.

In a mixed environment where you have some legacy and/or servers to manage I think using the hashicorp stack is a no brainer - consul and vault are tools I wouldn’t want to be without.

Post reply on HN