Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Nomad for a Home Server

mrkaran.dev

Post reply on HN
Print view View original

91–100 of 153 posts

Re: Running Nomad for a Home Server

#91

Post by GordonS »

3np wrote: post #52
marvinblum wrote: post #37

Earlier quoted context omitted.

I'm running a Traefik instance on each node, so that I can expose a service by adding a bunch of labels. The load balancer is not part of the cluster and routes the traffic to the nodes. You might want to consider that too :)

I'm doing exactly that, actually! Two bastion hosts/lbs sharing a virtual IP (keepalived), with two Traefik instances each (private and public). I actually schedule them through Nomad (on the host network interfaces) as well - since they solved the host networking issue I mentioned above it's properly set up with service checks. Super smooth to add and change services with consul catalog, and ACME-TLS included. Thing…

I recently setup Traefik 2.x to front a self-hosted Docker Registry, with automated Let's Encrypt renewals - I found the config to be really unintuitive and confusing! It feels like an awful lot of really finicky config for such a simply setup. Next time I'll try something else.

Re: Running Nomad for a Home Server

#92

Post by e12e »

cbushko wrote: post #55

> (Not joking) You are tired of running Helm charts or writing large YAML manifests. The config syntax for Nomad jobs is human friendly and easy to grasp. I write all of my kubernetes resources in terraform because I don't want to fight with helm charts. I was going to have to write something to monitor my deployments anyway and alert my co-workers that their deploys failed so why not just use terraform that tells yo…

Do you use terraform without helm (charts)? If not, don't you have to write charts from time to time?

Re: Running Nomad for a Home Server

#93

Post by dr_faustus »

I looked at Nomad about a year ago but deployment of a secure production cluster involved more moving parts than I liked. Especially the security seemed to be very complicated (with a private CA, certificates, Vault, etc. necessary). A simple shared secret would have sufficed imo, but that was not available as an option.

Re: Running Nomad for a Home Server

#94

Post by mads »

We used Nomad on Windows with the raw exec driver to schedule services on a project I worked on a year back. It worked great. Two binaries on each host along with a configuration file (for Nomad and Consul) and you are up and running.

Since then I have been a fan. Also worked a lot with Kubernetes, which has its merits, but the simplicity of Nomad is great.

Re: Running Nomad for a Home Server

#95

Post by chucky_z »

throwaway894345 wrote: post #62
marvinblum wrote: post #21

I did/do run both myself, Kubernetes and Nomad, and it was a million times easier to set up Nomad (including Consul) on bare metal than it was to set up Kubernetes. Kubernetes offers more features, but you most likely don't need them and the increase in complexity makes it a pain to maintain. I'm running a three-node cluster on Hetzner [0] for Pirsch [1] right now and haven't had any difficulties whatsoever when upgr…

I would be very interested in a more detailed write-up on Nomad vs Kubernetes for bare metal. I'm working through getting Kubernetes stood up, but I'm running into a dearth of features--namely you have to bring your own load balancer provider, storage provider, ingress controller, external DNS, monitoring, secret encryption, etc, etc before you can run any real world applications on top of it. I would be interested i…

Let me try to do some quick mapping...

> load balancer provider

consul connect handles this, how you get traffic to the ingresses is still DIY... kinda. you can also use consul catalog + traefik (I've actually put in some PRs myself to make traefik work with a really huge consul catalog so you can scale it to fronting thousands of services at once). there's also fabio. you can also get bgp ip injection with consul via https://github.com/mayuresh82/gocast run as a system job to get traffic to any LB (or any workload) if that's an option.

i've also ran haproxy and openresty without any problems getting stuff from consul catalog via nomad's template stanza and just signaling them on catalog changes.

> storage provider

anything CSI that doesn't have a 100% reliance on k8s works. if you're also just running docker underneath you can use anything compatible with docker volumes, like Portworx.

> ingress controller

consul connect ingress! or traefik, both kinda serve double duty here.

> external DNS

no good story here -- with one exception, if by "external" you mean "in the same DC but not the same host," consul provides a full DNS interface that we get a lot of mileage out of.

if you're managing everything with terraform though there's no reason you can't tie tf applies to route53/ns1/dyn or anything else though!

> monitoring

open up consul/nomad's prometheus settings and schedule vmagent on each node as a system job to scrape and dump somewhere. :)

we also use/have used/will use telegraf in some situations -- victoriametrics outright accepts influx protocol so you can do telegraf/vector => victoriametrics if you want to do that instead.

> secret encryption

this is all vault. don't be afraid of vault! vault is probably hashicorp's best product and it seems heavy but it's really not.

there's a lot here that doesn't really compare at all, like the exec/raw_exec drivers. we use those today to run some exotic workloads that do really poorly in containers or that have special networking needs that can map into containers but require a lot of extra operational effort, e.g.: glb-director and haproxy running GUE tunnels.

something interesting about the above is i'm testing putting the above in the same network namespace, so you can have containerized and non-containerized workloads in the same network cgroup namespace so you can share local networking across different task runners.

Re: Running Nomad for a Home Server

#96

Post by sweeneyrod »

ForHackernews wrote: post #71
linux2647 wrote: post #45

Earlier quoted context omitted.

I’ve often thought that when developers saw Kubernetes, what they actually wanted was some form of PaaS

Are there any lightweight but production-ready PaaS offerings out there? I know there's Dokku, https://flynn.io/ looked super promising but I think it's basically dead now, same for Deis that is dead and forked to https://web.teamhephy.com/ .

Capnrover seems like the best option. I have the same perception of Flynn. Dokku is nice but I don't really see how you could use it in production since it's limited to a single server (there are definitely some cases where that's all you need but I can't imagine you'd need a PaaS for most of them).

Re: Running Nomad for a Home Server

#97

Post by throwaway894345 »

dilyevsky wrote: post #89
throwaway894345 wrote: post #75

Earlier quoted context omitted.

Yeah, that's what I used. It comes with some providers out of the box, but they strike me as toys. For example, it gives you support for node-local volumes, but I don't really want to have to rely on my pods being scheduled on specific nodes (the nodes with the data). Even if you're okay with this, you still have to solve for data redundancy and/or backup yourself. The Rancher folks have a solution for this in the fo…

There’s metallb that lets you announce bgp to upstream routers. Another solution would be to just announce it via daemonset on every node and setup a nodeport. Or just add every frontend node IP into DNS. Obv all highly non-standard as it depends on your specific setup

Yes, to be clear, these problems can be worked around (although many such workarounds have their own tradeoffs that must be considered in the context of the rest of your stack as well as your application requirements); I was observing that the defaults are not what I would consider to be production-ready.

Re: Running Nomad for a Home Server

#98

Post by enobrev »

tylerjl wrote: post #61

I administer k8s daily at my full-time job and administer a 14-node nomad cluster in my homelab. This accurately captures my sentiments as well. My nomad cluster is even spread across three different CPU architectures (arm, arm64, and amd64) but still works great. One of the points I'd highlight in this post is just how good the combination of nomad, consul, and consul-template is. Even when nomad lacks some sort of…

I haven't used nomad yet, but I use consul and consul-template for configuration on a couple clusters (work and personal) and they're great.

Re: Running Nomad for a Home Server

#99

Post by jokethrowaway »

marvinblum wrote: post #21

I did/do run both myself, Kubernetes and Nomad, and it was a million times easier to set up Nomad (including Consul) on bare metal than it was to set up Kubernetes. Kubernetes offers more features, but you most likely don't need them and the increase in complexity makes it a pain to maintain. I'm running a three-node cluster on Hetzner [0] for Pirsch [1] right now and haven't had any difficulties whatsoever when upgr…

I had the opposite experience (in 2 different companies). Setting up K8s was quite straightforward and docs were helpful. We ended up building a deployment UI for it though.

Consul is nice and easy to use.

Nomad has been a painful experience: the default UI is confusing (people accidentally killed live containers), we have some small bits and pieces that don't quite behave as we expect and have no idea how to fix them. Error rate is too low to care and there are more pressing issues so likely WONTFIX. We often found ourselves looking into github issues for edge cases or over-allocating resources to overcome scheduling problems.

We considered just switching to their paid offering, just not to have to worry about this.

It kind of feels like that's their business model: attract engineers with OSS software and then upsell the paid version without all the warts.

Re: Running Nomad for a Home Server

#100

Post by cbushko »

e12e wrote: post #92
cbushko wrote: post #55

> (Not joking) You are tired of running Helm charts or writing large YAML manifests. The config syntax for Nomad jobs is human friendly and easy to grasp. I write all of my kubernetes resources in terraform because I don't want to fight with helm charts. I was going to have to write something to monitor my deployments anyway and alert my co-workers that their deploys failed so why not just use terraform that tells yo…

Do you use terraform without helm (charts)? If not, don't you have to write charts from time to time?

We do use some helm charts for the bigger things, gitlab runners, istio, thanos, prometheus, argo, etc. Some of those are run as directly from helm but many are being converted to use the terraform helm provider.

Our initial rollout on kubernetes had me writing about 30 helm charts for internal services. Once we saw helms shortcomings then they were converted to terraform. It was easy if you:

- helm template > main.yaml - use k2tf (https://github.com/sl1pm4t/k2tf) - some manual cleanup for inputs and such.

So now all of our product is terraformed, each as a module deployed to a namespace as an entire stack.

Post reply on HN