Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

161–170 of 369 posts

Re: Nomad vs. Kubernetes

#161

Post by brightball »

marvinblum wrote: post #33

I've been running a production-grade Nomad cluster on Hetzner for the past 1 1/2 years and it's fantastic. It was amazingly easy to set up compared to Kubernetes (which I also did), the UI is awesome, updates haven't broken anything yet (as long as you follow the changelog) and it's stable. I really like the separation of concerns the HashiStack offers. You can start out just using Consul for your service meshing, an…

I've been getting into Ansible and Terraform lately. It seems like Nomad would fit better in an environment where you weren't going to be putting everything in K8s, so a mixture where you have Consul and Vault handling standard VMs alongside your Nomad cluster would make a lot of sense.

Re: Nomad vs. Kubernetes

#163

Post by cogman10 »

LambdaComplex wrote: post #155
q3k wrote: post #43

Earlier quoted context omitted.

> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what K…

> the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is Could you expand on that? It sounds like an interesting position (bordering on the philosophical), but I don't know enough about Kubernetes to gauge its accuracy for myself.

k8s is built for extension with validation of the extensions you add ( https://kubernetes.io/docs/concepts/extend-kubernetes/api-ex... )

Helm is just sort of dumb text manipulation with a TINY bit of deployment management built on top of it. There isn't really a whole lot that helm buys over extending k8s.

Re: Nomad vs. Kubernetes

#164

Post by delta1 »

xyzzyz wrote: post #142
throwaway894345 wrote: post #108

Earlier quoted context omitted.

> kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} This is fine for the first deploy, but if you delete a resource from your manifests then kubectl doesn’t try to delete it from the cluster, so in practice you need something like Terraform (or perhaps ArgoCD) which actually tracks state. And of course as you mention, you probably want to generate these manifests to DRY up your YAML so you can actually main…

You can use -prune flag to apply to delete resources removed from your configs.

Which is still in alpha

Re: Nomad vs. Kubernetes

#165

Post by dividedbyzero »

throwaway923840 wrote: post #117
jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

They're both complex. But one of them has 10 times the components than the other, and requires you to use them. One of them is very difficult to install - so much so that there are a dozen different projects intended just to get it running. While the other is a single binary. And while one of them is built around containers (and all of the complexity that comes with interacting with them / between them), the other on…

Which is which?

Re: Nomad vs. Kubernetes

#166

Post by steeleduncan »

tetha wrote: post #150
Uberphallus wrote: post #124

Earlier quoted context omitted.

Hi, I see you mention the tiniest nodes in Hetzner there, whereas the Nomad documentation [0] talks about 3-5 server nodes in the 2-digit GiB memory range, which is what has kept me from trying Nomad as I find it insane. How much truth is there in the docs? [0] https://www.nomadproject.io/docs/install/production/requirem...

This very much depends on your workload, number of jobs and complexity of scheduling. Our Nomad servers have 4GB of memory in the VM and are using about 0.5 - 1G at a low three-digit number of jobs. Hashicorp is doing a smart but normal thing for on-prem sizing there - they are recommending specs which ensure you have no problems for a lot of workload sizes. And "workload" can grow very large there, since a single DC…

I think the minimum number of nodes is high because they are recommending the minimum requirements for a fault tolerant setup. It is entirely possible to install either k8s or nomad on a single node, but clearly that is not a fault tolerant situation.

IIRC both k8s and nomad rely on the Raft algorithm for consensus, and so both of them inherit the requirement of a minimum of 3 nodes from that algorithm.

If you want something to run some containers in the cloud, and you aren't concerned by the occasional failure, then there is no issue running a single Nomad (or k8s) on a single machine and later adding more if and when you require it.

Re: Nomad vs. Kubernetes

#167

Post by jaaron »

dividedbyzero wrote: post #165
throwaway923840 wrote: post #117

Earlier quoted context omitted.

They're both complex. But one of them has 10 times the components than the other, and requires you to use them. One of them is very difficult to install - so much so that there are a dozen different projects intended just to get it running. While the other is a single binary. And while one of them is built around containers (and all of the complexity that comes with interacting with them / between them), the other on…

Which is which?

From parent's comment:

k8s = 10x the components & difficult to install.

Nomad = single binary, works with but doesn't require containers.

Re: Nomad vs. Kubernetes

#168

Post by steveklabnik »

rkangel wrote: post #135
daenney wrote: post #123

These kinds of posts, by the vendor comparing to a competitor, always leave such a bad taste in my mouth. They decrease my confidence in both the product and the vendor. Stand on your own merits. There's a saying in Dutch: "Wij van Wc-eend adviseren Wc-eend"[1]. It basically boils down to pretending to give advice or information but you're just promoting your own interests. [1]: https://untranslatable.co/p/amarens/wi…

I know what mean, but it's not like they're pretending that much if it's an article hosted under nomadproject.io. The first question everyone is going to ask "why should I use this instead of K8s" so you might as well have a good answer. On the other hand, the Rust project has purposefully avoided "Rust vs X" comparisons on its website. I can't find the HN comments to back this up, but people like steveklabnik have i…

Yes. This topic is long and complicated; maybe I'll write it up or give a talk or something someday.

There is a lot of nuance.

Re: Nomad vs. Kubernetes

#169

Post by q3k »

LambdaComplex wrote: post #155
q3k wrote: post #43

Earlier quoted context omitted.

> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what K…

> the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is Could you expand on that? It sounds like an interesting position (bordering on the philosophical), but I don't know enough about Kubernetes to gauge its accuracy for myself.

There's two things:

1) Text templating YAML is just bad. It's the serialized format of some structured data - instead of templating its text representation (and dealing with quoting, nindent, stringly-typed variables, and general YAML badness), just manipulate the structures directly before serializing them. For example, write some Python code that composes these structures in memory and then just serializes them to YAML before passing it over to kubectl. You can also use ready made tooling that uses domain-specific languages well suited to manipulating and emitting such structured configuration, like jsonnet/kubecfg (or CUE/Dhall/Nix combined with kubectl apply -f). Eschewing text templating is not only more productive by letting you build abstractions (eg. to deal with Kuberenetes' verbosity with labels/selectors, etc.), it also allows you to actually compose more complex deployments together, like 'this wordpress library uses this mariadb library' or 'this mariadb library can take a list of objects that will be upserted as users on startup'.

2) Those YAML/JSON manifests aren't even that native to Kubernetes. A lot of things go behind the scenes to actually upsert a manifest, as Kubernetes' resource/object model isn't nearly as straightforward as 'apply this YAML document full of stuff' would indicate (there's state to mix in, API/schema changes, optional/default fields, changes/annotations by other systems...). With k8s' Server-Side-Apply this can now be fairly transparent and you can pretend this is the case, but earlier tooling definitely had to be smarter in order to apply changes. Things like doing structure-level diffs between the previously applied intent and the current intent and the current state in order to build a set of mutations to apply. What this means is that Helm's entire 'serialized as YAML, manipulated at text level' stage is not only harmful and a pain in the neck to work with (see 1.) but also unnecessary (as 'flat YAML file' isn't any kind of canonical, ready-to-use representation that Helm had to use).

Re: Nomad vs. Kubernetes

#170

Post by jaaron »

We're using Nomad for our backend at Singularity 6 (developing the mmo Palia).

In my experience (having worked with Nomad, k8s, mesos, and homegrown schedulers), k8s is fantastic from an early developer experience, but rough from an operations perspective. It's definitely gotten better, but there's a reason lots of folks want to have someone else host and operate their cluster.

Nomad is the opposite. Less tooling and bit more complexity for the app developer, significantly easier to operate at the cluster level. Nomad is also less "all-in" in it's architecture, which gives operators more flexibility in how they design and support the cluster.

The fact that we can run on anything and don't _have_ to use containers is a serious advantage, particularly for game development.

Post reply on HN