Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

191–200 of 369 posts

Re: Nomad vs. Kubernetes

#191

Post by gizdan »

wg0 wrote: post #179
specialp wrote: post #177

Earlier quoted context omitted.

It doesn't take long working with Nomad to hit the cases where you need to augment it. Now I know some of people enjoy being able to plug and play the various layers that you get in the complicated kitchen sink Kubernetes. We already had something that just ran containers and that was Mesos. They had the opinion that all the stuff like service discovery could be implemented and handled by other services like Marathon…

One of the things that I don't like about Nomad is HCL. It is a language that is mainly limited to HashiCorp tools and there's no wider adoption outside at least not to my knowledge. From the documentation: > Nomad HCL is parsed in the command line and sent to Nomad in JSON format via the HTTP API. So why not just JSON or even JSON at all and not MsgPack or just straight up HCL because that's over and over introduced…

JSON was designed for machine readability, HCL was designed for human readability.

HCL requires a lot more code to parse and many more resources to keep in memory vs JSON. I think it completely makes sense to do it this way. K8s is the same. On the server it does everything in JSON. Your YAML gets converted prior to sending to K8s.

Re: Nomad vs. Kubernetes

#192

Post by orthecreedence »

wg0 wrote: post #179
specialp wrote: post #177

Earlier quoted context omitted.

It doesn't take long working with Nomad to hit the cases where you need to augment it. Now I know some of people enjoy being able to plug and play the various layers that you get in the complicated kitchen sink Kubernetes. We already had something that just ran containers and that was Mesos. They had the opinion that all the stuff like service discovery could be implemented and handled by other services like Marathon…

One of the things that I don't like about Nomad is HCL. It is a language that is mainly limited to HashiCorp tools and there's no wider adoption outside at least not to my knowledge. From the documentation: > Nomad HCL is parsed in the command line and sent to Nomad in JSON format via the HTTP API. So why not just JSON or even JSON at all and not MsgPack or just straight up HCL because that's over and over introduced…

To my understanding, you can write most (all?) of the job/config files in JSON if you wish. At my company, we have a ton of HCL files because in the beginning it was easier to hand-write them that way, but we're now getting to the point where we're going to be templating them and going to switch to JSON. In other words, I believe HCL is optional.

Re: Nomad vs. Kubernetes

#193

Post by omneity »

tazjin wrote: post #2

As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't necessarily think that's for the better. Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" th…

Nomad was implemented based on the Borg paper.

Re: Nomad vs. Kubernetes

#194

Post by mkopinsky »

proxysna wrote: post #115
mkopinsky wrote: post #99

So far I've kept things simple, avoided k8s/HashiStack/etc by using docker compose with a simple docker-compose.yml for each server. This has been working well, but I'm starting to feel the pain points - HA requires copy-pasting yaml, I need to specify which services are on each server, and so on. What's the simplest next step I can take? I'd like something with (close to) the simplicity of docker compose, but ideall…

You should be ok with consul, ansible and templating your docker-compose files. Might take some time to set it all up, but should be ok.

Thanks for the suggestion.

Any suggested starting points for my research? DDG search for "ansible docker-compose" brings up some suggestions like [1] and [2] but I'm curious if you have other suggestions.

And just so I understand how these work together - I'd use Jenkins+ansible to push containers to my servers, I'd run consul within docker as well, and ... would Ansible register services with consul as it pushes them? Do the services need to be modified to speak with consul directly?

[1] https://docs.ansible.com/ansible/latest/collections/communit... [2] https://www.ansible.com/blog/six-ways-ansible-makes-docker-c...

Re: Nomad vs. Kubernetes

#195

Post by omneity »

jbotz wrote: post #95
proxysna wrote: post #11

Been running Nomad for a while now at work and home, and it is such a fun project to work and tinker with. Great community, lots of plugins and other great stuff. After running k8s in prod and home Nomad felt like a breath of fresh air in all aspects.

What do you do with Nomad at home? What comprises your hardware? Just curious...

I'm also using Nomad at home. It's a single node "cluster" in my desktop Ubuntu-based machine, really using it to deploy some self hosted services more easily (the usual suspects for home automation and the likes).

Re: Nomad vs. Kubernetes

#196

Post by p_l »

throwaway923840 wrote: post #133
q3k wrote: post #43

Earlier quoted context omitted.

> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what K…

> in fact, you probably shouldn't be using Helm Funny that the thing you shouldn't use is what the entire Kubernetes ecosystem uses for deployment. It's almost like there's no good way to do it.

Except it's... Not what the entire ecosystem uses. Yes, it's popular because it made for "app store"/"package repository" style operation, but I have yet to actually use it for longer than a few weeks despite running k8s projects across multiple companies since 2016.

Re: Nomad vs. Kubernetes

#197

Post by gizdan »

throwaway923840 wrote: post #117
jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

They're both complex. But one of them has 10 times the components than the other, and requires you to use them. One of them is very difficult to install - so much so that there are a dozen different projects intended just to get it running. While the other is a single binary. And while one of them is built around containers (and all of the complexity that comes with interacting with them / between them), the other on…

> But one of them has 10 times the components than the other

I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really?

Re: Nomad vs. Kubernetes

#198

Post by p_l »

sheeshkebab wrote: post #57
heinrichhartman wrote: post #41

Earlier quoted context omitted.

> How do you deploy a thing to run on k8s? kubectrl apply -f deploy.yaml should work, no? What forces you to use the sugar-coating?

Maybe bc no one uses that in reality and use helm instead?

I have yet to work in any environment where helm was used heavily and not just for few okne offs setup long ago and that now bring regrets.

Re: Nomad vs. Kubernetes

#199

Post by politician »

tetha wrote: post #172
steeleduncan wrote: post #166

Earlier quoted context omitted.

I think the minimum number of nodes is high because they are recommending the minimum requirements for a fault tolerant setup. It is entirely possible to install either k8s or nomad on a single node, but clearly that is not a fault tolerant situation. IIRC both k8s and nomad rely on the Raft algorithm for consensus, and so both of them inherit the requirement of a minimum of 3 nodes from that algorithm. If you want s…

Yes, I did not even consider that angle. And yes, nomad recommends 3 - 9 nodes in prod, maybe 11 or 13. The 3 are necessary to be able to tolerate one broken/maintained node and maintain the ability to schedule workloads. You can increase the number of tolerated nodes by increasing the number of nodes - 5 tolerate 2, 9 tolerate 3, 11 tolerate 5, 13 tolerate 6, but raft becomes slower with more nodes, because raft is…

PSA: In practice, choose 3 or 5 nodes. Going higher exposes system to increased communications overhead. See also: all of the literature on this.

Re: Nomad vs. Kubernetes

#200

Post by p_l »

clvx wrote: post #45
jnsaff2 wrote: post #34

I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built…

The worst part of k8s is definitely its configuration system. I don’t like helm as it’s combining packaging, templating and deployment lifecycle. I feel like each of this components should’ve been its own abstraction.

So just don't use Helm. It's not a core, crucial, or required part of k8s, it's just a reasonably popular related project... That I usually ban from environments before the rot can take root.
Post reply on HN