Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

261–270 of 369 posts

Re: Nomad vs. Kubernetes

#261

Post by zmmmmm »

johnnycerberus wrote: post #59
EdwardDiego wrote: post #56

Earlier quoted context omitted.

Plenty of K8s operators written in Java. Sure, it's not not underpinning K8s itself, but tbh, what language K8s is written in doesn't really matter, so long as it works.

Yes, exactly my thoughts, the problem is that devops generally are used to Python/Go and in my company though they don't make it mandatory, they recommend Go. Also, they have a repulsive reaction to everything .NET and JVM :).

> Also, they have a repulsive reaction to everything .NET and JVM

It's an interesting phenomenon I observe quite commonly.

I think in the devops space they see these VM based languages as basically introducing ten redundant layers of unnecessary assumptions and complexity on top of what is already a good foundation - the Unix OS layer. They know the unix os layer well but every time they deploy one of these VM based languages it creates headaches through unique and strange behaviour that violates their assumptions and knowledge and they have no way to learn it since you need years of experience as a developer to become comfortable with it all. From a developer perspective, we see the OS as this annoying ball of complexity and want VM languages to manage that and make it go away.

So it's all about what you know better and where your comfort zone is in the end. For a long time the developers had the upper hand because the OS story was a train wreck, but containers have turned that around in recent years, so now it is not such a crazy thing to pin down the exact OS and version and entire dependency chain you are going to deploy on.

Re: Nomad vs. Kubernetes

#262

Post by cpuguy83 »

EdwardDiego wrote: post #54
_hzrk wrote: post #5

Earlier quoted context omitted.

> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. I know that Borg was written in Java and Kubernetes in Go. Though the latter had a reputation in the beginning as a systems programming language, its purpose was actually to build large-scale cloud infrastructure projects with it and it proved formidably well suited for the task. It compiles fast, anyon…

I just wish klog would go die in a fire, I'm so confused at why logging is so terrible in Go.

klog is the descendant of glog which is a go implementation of Google's logging. Go has nothing to do with it except it's the logging that k8s took on (originally it used glog).

-- edit typo

Re: Nomad vs. Kubernetes

#263

Post by tetha »

martijn9612 wrote: post #218
chucky_z wrote: post #205

Earlier quoted context omitted.

I am currently running over 13000 nodes with a 5 server cluster. These are large boxes (90gb memory, 64 cores) but cpu usage is under 20% and memory usage is under 30%.

I'm sorry, what is the exact benefit of running multiple nodes on one piece of hardware? Just software failure resilience?

I don't see where you see multiple nodes. I'd read the parent post as: There are 5 identically sized, beefy VMs. Each of these 5 beefy VMs run 1 instance of the nomad server. And then, there is 13k other VMs or physical systems, which run nomad in client mode. These clients connect to the server to register and get allocated allocations / task groups / practically containers or vms or binaries to run.

We're nowhere near that size, but the architecture is probably identical.

Re: Nomad vs. Kubernetes

#264

Post by sofixa »

glynnforrest wrote: post #61

> Flexible Workload Support This is Nomad's most underrated feature, IMHO. You don't have to use containers for everything if you don't want to. For example, if you're a golang shop you can run everything as native binaries and cut out docker completely. Nomad has much simpler networking, i.e. no web of iptables rules to figure out. You can add Consul connect as a service mesh if you need it, but if you don't, you ca…

> The main downside for me is a lack of plug and play pieces,

Hashicorp just announced Nomad Packs filling that precise niche. Still in beta, and it was a long time coming but IMHO it was the main thing missing and is honestly awesome.

Re: Nomad vs. Kubernetes

#265

Post by chucky_z »

alexeldeib wrote: post #257
martijn9612 wrote: post #218

Earlier quoted context omitted.

I'm sorry, what is the exact benefit of running multiple nodes on one piece of hardware? Just software failure resilience?

I assume they meant a 5 server control plane supporting 13k worker nodes, not that they partitioned 5 large hosts into 13k smaller ones. It's a counterpoint to GP's "raft gets slow with more servers", I think.

This is correct.

We run a Raft cluster of 5 voting members with 13,000 physical Nomad nodes mixed across OS's with a bunch of random workloads using docker, exec, raw_exec, java, and some in-house drivers. I'll clarify that (and I can't... because I can't edit my post anymore :( ).

Re: Nomad vs. Kubernetes

#266

Post by sofixa »

gizdan wrote: post #244
sofixa wrote: post #235

Earlier quoted context omitted.

That's false. Vault has integrated storage and no longer needs Consul. If you want to have the Enterprise versions( which aren't required), you just need 1 each of Nomad, Consul, Vault. Considering many people use Vault with Kubernetes anyway(due to the joke that is Kubernetes "secrets"), and Consul provides some very nice features and is quite popular itself, that's okay IMHO. Unix philosophy and all.

This is just false. I've run Vault in an Enterprise and unless something has changed in the last 12 months, Hashicorp's recommendation for Vaul has been 1 Consul cluster for Vault's data store, and 1 for it's (and other application's) service discovery. Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.

It has been longer than 12 months that Vault has had integrated storage.

Re: Nomad vs. Kubernetes

#267

Post by sofixa »

alexeldeib wrote: post #254
gizdan wrote: post #244

Earlier quoted context omitted.

This is just false. I've run Vault in an Enterprise and unless something has changed in the last 12 months, Hashicorp's recommendation for Vaul has been 1 Consul cluster for Vault's data store, and 1 for it's (and other application's) service discovery. Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.

https://www.vaultproject.io/docs/configuration/storage/raft It's new but I think is quickly becoming preferred. I found trying to setup nomad/consul/vault as described on the hashi docs creates some circular dependencies tbh (e.g. the steps to setup nomad reference a consul setup, the steps for vault mention nomad integration, but there's no clear path outside the dev server examples of getting there without reading…

It's not that circular - you start with Consul, add Vault and then Nomad, clustering them through Consul and configuring Nomad to use Vault and Consul for secrets and KV/SD respectively. And of course it can be done incrementally ( you can deploy Nomad without pointing it to Consul or Vault, and just adding that configuration later).

Re: Nomad vs. Kubernetes

#268

Post by InvertedRhodium »

k8sToGo wrote: post #259
zie wrote: post #249

Earlier quoted context omitted.

Sure, you pay someone else to keep k8s alive, it's not so bad, but it's expensive to do that. You generally need a full-time team of people to keep a k8s deployment alive if you are running it yourself. I keep Nomad alive part-time. It's operationally simple and easy to wrap ones head around it and understand how it works.

You can also use something like Rancher or k3s to keep it alive part-time.

RKE2 is the closest thing we've found to a turnkey, on prem, production ready Kubernetes distribution. Though we were last looking about a year ago so I concede there might be better or comparable options now.

Re: Nomad vs. Kubernetes

#269

Post by gizdan »

alexeldeib wrote: post #254
gizdan wrote: post #244

Earlier quoted context omitted.

This is just false. I've run Vault in an Enterprise and unless something has changed in the last 12 months, Hashicorp's recommendation for Vaul has been 1 Consul cluster for Vault's data store, and 1 for it's (and other application's) service discovery. Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.

https://www.vaultproject.io/docs/configuration/storage/raft It's new but I think is quickly becoming preferred. I found trying to setup nomad/consul/vault as described on the hashi docs creates some circular dependencies tbh (e.g. the steps to setup nomad reference a consul setup, the steps for vault mention nomad integration, but there's no clear path outside the dev server examples of getting there without reading…

Cool, so that's certainly new. But even then, you're dealing with the Raft protocol. The different is it's built into Nomad compared to Kubernetes where it's a separate service. I just don't see Nomad and Co being that much easier to run, if at all.

I think Nomad's biggest selling point is that it can run more than just containers. I'm still not convince that's it's much better. At best it's equal.

Re: Nomad vs. Kubernetes

#270

Post by alexeldeib »

gizdan wrote: post #269
alexeldeib wrote: post #254

Earlier quoted context omitted.

https://www.vaultproject.io/docs/configuration/storage/raft It's new but I think is quickly becoming preferred. I found trying to setup nomad/consul/vault as described on the hashi docs creates some circular dependencies tbh (e.g. the steps to setup nomad reference a consul setup, the steps for vault mention nomad integration, but there's no clear path outside the dev server examples of getting there without reading…

Cool, so that's certainly new. But even then, you're dealing with the Raft protocol. The different is it's built into Nomad compared to Kubernetes where it's a separate service. I just don't see Nomad and Co being that much easier to run, if at all. I think Nomad's biggest selling point is that it can run more than just containers. I'm still not convince that's it's much better. At best it's equal.

> you're dealing with the Raft protocol. The different is it's built into Nomad compared to Kubernetes where it's a separate service

I don't really follow this. etcd uses raft for consensus, yes, and it's built in. Kubernetes components don't use raft across independent services. Etcd is the only component that requires consensus through raft. In hashi stack, vault and nomad (at least) both require consensus through raft. So the effect is much bigger in that sense.

> I think Nomad's biggest selling point is that it can run more than just containers. I'm still not convince that's it's much better. At best it's equal.

Totally agree. The driver model was very forward looking compared to k8s. CRDs help, but it's putting a square peg in a round hole when you want to swap out Pods/containers.

Post reply on HN