Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Nomad for a Home Server

mrkaran.dev

Post reply on HN
Print view View original

71–80 of 153 posts

Re: Running Nomad for a Home Server

#71

Post by ForHackernews »

linux2647 wrote: post #45
mrweasel wrote: post #22

Maybe it's because Nomad is a commercial product as well, but it's really sad that it doesn't get more usage. Kubernetes is the new hotness, to the point where people don't care that the don't really need Kubernetes. They just know that the big boys are using it, so regardless of scale, so should they. Kubernetes is exceedingly complex, and it needs to be, to support all it's use cases. However, I would claim that th…

I’ve often thought that when developers saw Kubernetes, what they actually wanted was some form of PaaS

Are there any lightweight but production-ready PaaS offerings out there?

I know there's Dokku, https://flynn.io/ looked super promising but I think it's basically dead now, same for Deis that is dead and forked to https://web.teamhephy.com/.

Re: Running Nomad for a Home Server

#72

Post by sofixa »

throwaway894345 wrote: post #62
marvinblum wrote: post #21

I did/do run both myself, Kubernetes and Nomad, and it was a million times easier to set up Nomad (including Consul) on bare metal than it was to set up Kubernetes. Kubernetes offers more features, but you most likely don't need them and the increase in complexity makes it a pain to maintain. I'm running a three-node cluster on Hetzner [0] for Pirsch [1] right now and haven't had any difficulties whatsoever when upgr…

I would be very interested in a more detailed write-up on Nomad vs Kubernetes for bare metal. I'm working through getting Kubernetes stood up, but I'm running into a dearth of features--namely you have to bring your own load balancer provider, storage provider, ingress controller, external DNS, monitoring, secret encryption, etc, etc before you can run any real world applications on top of it. I would be interested i…

I'm currently working on an article exploring Nomad and how it compares in some aspects with Kubernetes, which i'll post on HN soon-ish.

Re: Running Nomad for a Home Server

#73

Post by sofixa »

omneity wrote: post #46
ericbarrett wrote: post #40

Earlier quoted context omitted.

The selling point of GKE etc. is “minimal to no maintenance,” but of course somebody else is doing the maintenance and the customer is paying a premium for it. Says great things about Nomad.

Yeah, when making the decision it was quite harrowing to think of maintaining a cluster in production. Nomad had very little operational complexity compared to what we imagined. We've had two main outages in months: - Server disks were filling up and we hadn't set up monitoring properly at the time (ironic for the name of our company :) ). Not Nomad's fault. - A faulty healthcheck caused all the servers of a cluster…

Jaeger is pretty great for tracing, and can integrate with Traefik/Envoy ( or whatever you use for ingress/inter-service communication).

We're running Loki for the logs ( via nomad log forwared/shipper and promtail) and so far it's going great. I'll have to do a write-up about the the whole thing.

Re: Running Nomad for a Home Server

#74

Post by ddreier »

OJFord wrote: post #68

> - Job: Job is a collection of different groups. Job is where the constraints for type of scheduler, update strategies and ACL is placed. > - Group: Group is a collection of different tasks. A group is always executed on the same Nomad client node. You'll want to use Groups for use-cases like a logging sidecar, reverse proxies etc. > - Task: Atomic unit of work. A task in Nomad can be running a container/binary/Java…

I think your description is more accurate.

A Task is an individual unit of work: a container, executable, JVM app, etc.

A Task Group/Allocation is a group of Tasks that will be placed together on nodes as a unit.

A Job is a collection of Task Groups and is where you tell Nomad how to schedule the work. A "system" Job is scheduled across all Nodes somewhat like a DaemonSet. A "batch" Job can be submitted as a one-off execution, periodic (run on a schedule) or parameterized (run on demand with variables specified at runtime). A "service" Job is the "normal" scheduler where all Task Groups are treated like ReplicaSets.

Placement is determined by constraints that can be specified on each Task Group.

Re: Running Nomad for a Home Server

#75

Post by throwaway894345 »

ForHackernews wrote: post #70
throwaway894345 wrote: post #62

Earlier quoted context omitted.

I would be very interested in a more detailed write-up on Nomad vs Kubernetes for bare metal. I'm working through getting Kubernetes stood up, but I'm running into a dearth of features--namely you have to bring your own load balancer provider, storage provider, ingress controller, external DNS, monitoring, secret encryption, etc, etc before you can run any real world applications on top of it. I would be interested i…

I'd be really interested, too. Have you looked at k3s at all? We're considering trying to run https://github.com/rancher/k3os on bare metal servers.

Yeah, that's what I used. It comes with some providers out of the box, but they strike me as toys. For example, it gives you support for node-local volumes, but I don't really want to have to rely on my pods being scheduled on specific nodes (the nodes with the data). Even if you're okay with this, you still have to solve for data redundancy and/or backup yourself. The Rancher folks have a solution for this in the form of LongHorn, so maybe we can expect that to be integrated into k3s in the future. There's also no external DNS support at all, and IIRC the default load-balancer provider (Klipper LB, which itself seems to be not very well documented) assigns node IPs to services (at random, as far as I can tell) so it's difficult to bind a DNS name to a service without something dynamically updating the records whenever k8s changes the service's external IP address (and even then, this is not a recipe for "high availability" since the DNS caches will be out of date for some period of time). Basically k8s is still immature for bare metal; distributions will catch up in time, but for now a lot of the hype outpaces reality.

Re: Running Nomad for a Home Server

#76

Post by 4eor0 »

superkuh wrote: post #50
4eor0 wrote: post #44

Earlier quoted context omitted.

It’s software? It’s all of triggering an ordered copy-paste process (downloading and installing is just a less random copy-paste). I dunno about you but I don’t download Ubuntu’s package repo. I have to run an install command and then customize nginx. What’s the real logistical difference to the user if they run this command or that command? Or set config values in this file or that? Why are you using Linux at home?…

As the article says, >Nomad is also a simpler piece to keep in your tech stack. Sometimes it's best to keep things simple when you don't really achieve any benefits from the complexity. Simple is better. But in this case he doesn't realize that he's stuck way up the complexity stack in a local minimum that's way more complex than most of the potential software landscape. None of that is needed to expose a webserver.…

Yes I get how it works in the base case.

What we disagree on is the level of complexity.

It’s cli commands and text editing regardless of which method is used.

There is no iron clad hierarchy for organizing computers files on top of the OS, only personal experience. What may seem more complex for you is still just cli commands and text for me.

I do not see config as having weight, heft. Data, sure. Config is arbitrary.

Re: Running Nomad for a Home Server

#77

Post by qbasic_forever »

rektide wrote: post #67

A whole lot of anti- Kubernetes "you most likely don't need them and the increase in complexity makes it a pain to maintain" and "Kubernetes is exceedingly complex" in this thread & somewhat in this article. I agree that you probably don't need Kubernetes, and perhaps yeah it could be considered complex. But I think it's the right fit for most developers/doers & over time most operators too. Kubernetes is not Kuberne…

Yeah if there were a one-click install _complete_ local distribution of Kubernetes with a GUI, and that just read and migrated all your docker-compose.yaml files I think it would see much less complaints and teeth-gnashing from new folks. The existing tools like minikube, kind, microk8s, are still too clunky and don't include everything you'd want for a good local setup (login/authentication, ingress, registry, easy mounting local host volumes, a good operations workflow with gitops primitives, etc.). Docker Desktop comes close with a one click turn on a kubernetes cluster, but the handwriting is on the wall for the end of docker & kubernetes tight integration. All of the pieces are out there to make a buttery smooth and slick local Kubernetes stack but unfortunately you have to seek them out and kludge it all together yourself.

But if you do feel ambitious, k8s + flux gitops toolkit + tekton CI/CD + knative serving & eventing + skaffold is one heck of a productive and amazing stack for development (and for bonus points switch your code to a bazel monorepo and rules_k8s for another awesome experience).

Re: Running Nomad for a Home Server

#78

Post by mr-karan »

rektide wrote: post #67

A whole lot of anti- Kubernetes "you most likely don't need them and the increase in complexity makes it a pain to maintain" and "Kubernetes is exceedingly complex" in this thread & somewhat in this article. I agree that you probably don't need Kubernetes, and perhaps yeah it could be considered complex. But I think it's the right fit for most developers/doers & over time most operators too. Kubernetes is not Kuberne…

Author here. I explicitly mentioned about the Operator Pattern in my post and why K8s probably makes more sense in this context :)

Re: Running Nomad for a Home Server

#79

Post by mr-karan »

OJFord wrote: post #68

> - Job: Job is a collection of different groups. Job is where the constraints for type of scheduler, update strategies and ACL is placed. > - Group: Group is a collection of different tasks. A group is always executed on the same Nomad client node. You'll want to use Groups for use-cases like a logging sidecar, reverse proxies etc. > - Task: Atomic unit of work. A task in Nomad can be running a container/binary/Java…

Author here. Hm, now that you point this out, I do get your point. However you can't run multiple different Pods in a deployment (you can run multiple containers in a Pod), that's why a Job isn't really comparable to a ReplicaSet.

I could very well be wrong, but this is my understanding.

Re: Running Nomad for a Home Server

#80

Post by ForHackernews »

throwaway894345 wrote: post #75
ForHackernews wrote: post #70

Earlier quoted context omitted.

I'd be really interested, too. Have you looked at k3s at all? We're considering trying to run https://github.com/rancher/k3os on bare metal servers.

Yeah, that's what I used. It comes with some providers out of the box, but they strike me as toys. For example, it gives you support for node-local volumes, but I don't really want to have to rely on my pods being scheduled on specific nodes (the nodes with the data). Even if you're okay with this, you still have to solve for data redundancy and/or backup yourself. The Rancher folks have a solution for this in the fo…

I wonder how their Rio thing stacks up? https://rancher.com/blog/2019/introducing-rio
Post reply on HN