Nomad vs. Kubernetes • Hacker News Board

Re: Nomad vs. Kubernetes

#201

Post by rkangel » Fri, Oct 15, 2021, 4:59 PM UTC

steveklabnik wrote: post #168

rkangel wrote: post #135

Earlier quoted context omitted.

I know what mean, but it's not like they're pretending that much if it's an article hosted under nomadproject.io. The first question everyone is going to ask "why should I use this instead of K8s" so you might as well have a good answer. On the other hand, the Rust project has purposefully avoided "Rust vs X" comparisons on its website. I can't find the HN comments to back this up, but people like steveklabnik have i…

Yes. This topic is long and complicated; maybe I'll write it up or give a talk or something someday. There is a lot of nuance.

I'd love to read a blog post on that. I completely see how it's a difficult line to walk - you don't want to be "Rust is better than all these rubbish languages" but you still want to provide people with information so that they can make informed choice.

Re: Nomad vs. Kubernetes

#202

Post by shrumm » Fri, Oct 15, 2021, 4:59 PM UTC

To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes.

All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself.

With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model.

Re: Nomad vs. Kubernetes

#203

Post by jeffbee » Fri, Oct 15, 2021, 5:02 PM UTC

dilyevsky wrote: post #186

jeffbee wrote: post #149

Earlier quoted context omitted.

Borg is what k8s could be if there were any kind of ground rules and people were willing to call out some use cases as being legitimately stupid. Compared to k8s, Borg is a model of usability and simplicity.

Ground rules as in “you have to compile with 100s of these google3 libraries to run a simple app”?

I don't think that's really true at all. You can pretty easily run any damned thing in Borg. If you want it to do stuff with other Google services then you need a lot of Google code, but if you just want to run 100000 replicas of `/bin/sleep 100000`, Borg makes it easy.

Re: Nomad vs. Kubernetes

#204

Post by siliconc0w » Fri, Oct 15, 2021, 5:06 PM UTC

What is nice about using the same tech as the 'big players' is you get the benefits of massive armies of engineers building integrated products and features to sell to other big users of the product. This means there are options to add on distributed databases, storage, monitoring, tracing, CD/CI, etc. So it can be worth swallowing 'big player' complexity if it means you can play in the same ecosystem. If you are already on a cloud then your infrastructure provider will handle a lot of that complexity for you anyway and it's often better integrated into their other products (i.e networking, access control, etc).

Re: Nomad vs. Kubernetes

#205

Post by chucky_z » Fri, Oct 15, 2021, 5:10 PM UTC

tetha wrote: post #172

steeleduncan wrote: post #166

Earlier quoted context omitted.

I think the minimum number of nodes is high because they are recommending the minimum requirements for a fault tolerant setup. It is entirely possible to install either k8s or nomad on a single node, but clearly that is not a fault tolerant situation. IIRC both k8s and nomad rely on the Raft algorithm for consensus, and so both of them inherit the requirement of a minimum of 3 nodes from that algorithm. If you want s…

Yes, I did not even consider that angle. And yes, nomad recommends 3 - 9 nodes in prod, maybe 11 or 13. The 3 are necessary to be able to tolerate one broken/maintained node and maintain the ability to schedule workloads. You can increase the number of tolerated nodes by increasing the number of nodes - 5 tolerate 2, 9 tolerate 3, 11 tolerate 5, 13 tolerate 6, but raft becomes slower with more nodes, because raft is…

I am currently running over 13000 nodes with a 5 server cluster. These are large boxes (90gb memory, 64 cores) but cpu usage is under 20% and memory usage is under 30%.

Re: Nomad vs. Kubernetes

#206

Post by p_l » Fri, Oct 15, 2021, 5:10 PM UTC

jeffbee wrote: post #203

dilyevsky wrote: post #186

Earlier quoted context omitted.

Ground rules as in “you have to compile with 100s of these google3 libraries to run a simple app”?

I don't think that's really true at all. You can pretty easily run any damned thing in Borg. If you want it to do stuff with other Google services then you need a lot of Google code, but if you just want to run 100000 replicas of `/bin/sleep 100000`, Borg makes it easy.

Because from what I heard, all the other crucial to me benefits of k8s are provided through linking of lots and lots of Google internal libraries?

Re: Nomad vs. Kubernetes

#207

Post by chucky_z » Fri, Oct 15, 2021, 5:11 PM UTC

If you have a mixed Windows/Linux environment I cannot suggest Nomad strongly enough. It truly shines in this environment. Especially if you're dealing with legacy stuff.

Even better if you're running Mac + BSD as well, it just runs everywhere.

Re: Nomad vs. Kubernetes

#208

Post by orthecreedence » Fri, Oct 15, 2021, 5:14 PM UTC

maccard wrote: post #85

mrweasel wrote: post #15

There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? Is it simply a matter of Kubernetes being an open source project and Nomad being owned by HashiCorp?

I can't run nomad on aws without managing a nomad cluster myself. I _can_ do that with k8s.

How have you found the managed k8s on AWS? I tend to shy away from their managed services until it has gone through a 4-5 year grace period, interested to know what your experience has been.

Re: Nomad vs. Kubernetes

#209

Post by dilyevsky » Fri, Oct 15, 2021, 5:15 PM UTC

jeffbee wrote: post #203

dilyevsky wrote: post #186

Earlier quoted context omitted.

Ground rules as in “you have to compile with 100s of these google3 libraries to run a simple app”?

I don't think that's really true at all. You can pretty easily run any damned thing in Borg. If you want it to do stuff with other Google services then you need a lot of Google code, but if you just want to run 100000 replicas of `/bin/sleep 100000`, Borg makes it easy.

Sure but none of the stuff like authentication and access to any google systems will work. Kubernetes is complex bc it can take damn near any docker container and it can run seamlessly

Re: Nomad vs. Kubernetes

#210

Post by raffraffraff » Fri, Oct 15, 2021, 5:16 PM UTC

gizdan wrote: post #197

throwaway923840 wrote: post #117

Earlier quoted context omitted.

They're both complex. But one of them has 10 times the components than the other, and requires you to use them. One of them is very difficult to install - so much so that there are a dozen different projects intended just to get it running. While the other is a single binary. And while one of them is built around containers (and all of the complexity that comes with interacting with them / between them), the other on…

> But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did y…

Also, Kubernetes can be just a single binary if you use k0s or k3s. And if you don't want to run it yourself you can use a managed k8s from AWS, Google, Digital Ocean, Oracle...