Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad v1.0 release – workload orchestration

github.com

Post reply on HN
Print view View original

21–27 of 27 posts

Re: Nomad v1.0 release – workload orchestration

#21

Post by tpetry »

ngrilly wrote: post #17
schmichael wrote: post #7

Earlier quoted context omitted.

Thanks that's great feedback and you're not alone in that desire. We do "small" users a pretty big disservice by effectively dismissing single-server deployments and jumping straight to "real" production deployments (3+ servers distinct from the nodes that run workloads). We have people who use Nomad locally as a systemd alternative. We have people who use single-scheduler-multiple-nodes clusters at home or at work,…

I’d be happy to know more about the single node use case. I considered Nomad in the past for this, but it seemed optimized for a cluster, and I ended up using Docker Swarm instead.

Same for me, i did stick to docker-compose. A production-grade single server deployment would be a real interesting case.

Re: Nomad v1.0 release – workload orchestration

#23

Post by big-malloc »

ngrilly wrote: post #18
schmichael wrote: post #7

Earlier quoted context omitted.

Thanks that's great feedback and you're not alone in that desire. We do "small" users a pretty big disservice by effectively dismissing single-server deployments and jumping straight to "real" production deployments (3+ servers distinct from the nodes that run workloads). We have people who use Nomad locally as a systemd alternative. We have people who use single-scheduler-multiple-nodes clusters at home or at work,…

For single node or small cluster deployments, one issue with k8s is the significant CPU and RAM usage even on a fully idle system with no payload. How is Nomad in that regard? Are the controllers also implemented as a control loop that is constantly polling and diffing reported and desired state?

Granted this is anecdotal, but I found Nomad to have much more reasonable resource usage compared to K3s on my pi cluster

Re: Nomad v1.0 release – workload orchestration

#24

Post by ngrilly »

big-malloc wrote: post #23
ngrilly wrote: post #18

Earlier quoted context omitted.

For single node or small cluster deployments, one issue with k8s is the significant CPU and RAM usage even on a fully idle system with no payload. How is Nomad in that regard? Are the controllers also implemented as a control loop that is constantly polling and diffing reported and desired state?

Granted this is anecdotal, but I found Nomad to have much more reasonable resource usage compared to K3s on my pi cluster

Thanks. Seems like it’s worth looking into it then. Still curious about the reasons why it’s more efficient.

Re: Nomad v1.0 release – workload orchestration

#25

Post by ngrilly »

tpetry wrote: post #21
ngrilly wrote: post #17

Earlier quoted context omitted.

I’d be happy to know more about the single node use case. I considered Nomad in the past for this, but it seemed optimized for a cluster, and I ended up using Docker Swarm instead.

Same for me, i did stick to docker-compose. A production-grade single server deployment would be a real interesting case.

The big problem for me with Docker Compose is the lack of zero downtime / seamless deployments. First it stops the containers running the old version, then it starts new containers running new versions, instead of doing the reverse.

Re: Nomad v1.0 release – workload orchestration

#26

Post by schmichael »

ngrilly wrote: post #24
big-malloc wrote: post #23

Earlier quoted context omitted.

Granted this is anecdotal, but I found Nomad to have much more reasonable resource usage compared to K3s on my pi cluster

Thanks. Seems like it’s worth looking into it then. Still curious about the reasons why it’s more efficient.

Nomad was designed for efficiency (linear scalability) and performance from the very beginning. One of major aspects of this is that it is monolithic. The single nomad binary contains:

- The CLI for interacting with Nomad clusters via their HTTP API

- The HTTP API

- The "server" agent which contains the scheduler, state store, and Raft implementation

- The "client" agent which runs workloads and has builtin task drivers for docker, plain processes, and more.

Not only is there no dependency on an external state store like etcd, but there's no pluggable abstraction for state storage. The tradeoff is that all cluster state must fit in memory. Pi clusters should use far less than 100mb of memory while something like C2M (6,100 nodes, 2 million containers) used just under 100gb of memory. Memory use scales linearly with cluster size (nodes+containers).

Since Raft and the scheduler are colocated within a single server agent process, many operations can be performed in the replicated FSM as a simple function call instead of if Raft was an external process requiring networking overhead to interact with.

I'm not very familiar with k8s internals, so I'm afraid I can't offer a very detailed direct comparison.

Re: Nomad v1.0 release – workload orchestration

#27

Post by ngrilly »

schmichael wrote: post #26
ngrilly wrote: post #24

Earlier quoted context omitted.

Thanks. Seems like it’s worth looking into it then. Still curious about the reasons why it’s more efficient.

Nomad was designed for efficiency (linear scalability) and performance from the very beginning. One of major aspects of this is that it is monolithic. The single nomad binary contains: - The CLI for interacting with Nomad clusters via their HTTP API - The HTTP API - The "server" agent which contains the scheduler, state store, and Raft implementation - The "client" agent which runs workloads and has builtin task driv…

Thanks for that detailed explanation. The fact that all main functions live in a single process probably helps a lot with CPU and RAM usage, compared to k8s where etcd, API server, scheduler and controller manager live in independent processes, which means serialization/deserialization and less optimization opportunities.
Post reply on HN