Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad, a cluster manager and scheduler

hashicorp.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Nomad, a cluster manager and scheduler

#41

Post by ninkendo »

fapjacks wrote: post #36
SEJeff wrote: post #33

I'm genuinely not sure why they are trying to compete with the likes of Mesos or Kubernetes, or what they are really trying to achieve here. There is simply no way they'll build a community around nomad 1/2 as big as either of the aforementioned even if the software is really good.

I'm sure to a fan that "really freaking loves Mesos" you can't imagine other software competing with Mesos, but Hashicorp writes excellent software generally. Serf and Consul have completely sold me on Nomad, and I haven't even used it yet. I think there will be similar draw for many people that have used Hashicorp's software before.

It's just sad that there's yet another player that wants to be at the very bottom of the clustering stack. For physical resource management (memory and CPUs, etc), by definition you have to pick one solution and stick with it (because you don't want multiple cluster managers thinking they exclusively own the same hardware.)

Mesos is the only player right now that's trying to do multitenancy; separating the concerns of resource management from scheduler algorithms, UI's, etc. You put mesos on your hardware as your datacenter kernel, then you play around with various schedulers to support different workloads.

Done properly, I'd much rather see Nomad exist as a Mesos framework, so I could colocate it along side chronos/spark/marathon and see how it fits in. Instead if I wanted to run this I have to partition up my hardware and dedicate some to it, which is the whole thing that cluster managers are supposed to save you from having to do (by running multiple workloads on the same hardware.)

I get that this is operationally simpler to set up and install, but with that simplicity comes a huge amount of lock-in, because you can't do anything else with your hardware other than run Nomad on it.

Re: Nomad, a cluster manager and scheduler

#42

Post by lobster_johnson »

SEJeff wrote: post #40
atombender wrote: post #38

Earlier quoted context omitted.

This is a space where you can compete on operational complexity. Mesos and Kubernetes are fairly complicated to set up and run, with dependencies and intricacies that Nomad just doesn't have. From your other comment: > I do find it ironic that they talk about how nomad is for microservices and then make a dig at the several microservices that k8s is made up of. Being for microservices doesn't mean you should be a mic…

When they become bottlenecks, you scale them out. That is the beauty of the design, vs a monolith like nomad :) It does win on setup for sure. This isn't an easy problem to solve however.

Not all bottlenecks scale linearly (and "bottleneck" was just one of several words I used). I'm sure Kubernetes is well-designed and battle-tested, but to most developers these are black boxes that need to be studied and learned, each with its own set of complexities and workarounds and warts. And so on.

Moving parts are moving parts no matter how well they are designed. More of them always add complexity, by definition; they never reduce it.

Re: Nomad, a cluster manager and scheduler

#43

Post by smarterclayton »

illamint wrote: post #32
rgarcia wrote: post #30

Earlier quoted context omitted.

Also, their messaging seems a little disingenuous. Otto talks about how important it is to support microservice development and deployment, but Nomad lists as a con that Kubernetes has too many separately deployed and composed services. This is consistent with a (reasonable) belief that microservice architecture is an important design pattern to support, but may not be the best approach for all problems. From reading…

I think there's definitely a bootstrapping problem here: microservices are great if you have something like Kubernetes, Nomad, Mesos etc. on which to run and deploy them, but you have to run your platform on something and be able to bring it back up if it goes down and that's where I think Nomad might have the edge.

Agree (Kubernetes and OpenShift dev here) - OpenShift is actually bundled as a monolithic Go binary that contains the full Kubernetes stack and client, the Openshift admin client, user client, and js web console for exactly that reason (even though it is all technically micro services on the server side). The single binary comes with downsides (binary is 95M) but it makes the "try it out" flow much, much, easier to see it all working. But the converse is true - you have to be able to decouple those bits at scale, and you eventually will want to start leveraging the platform to run itself.

Re: Nomad, a cluster manager and scheduler

#44

Post by krenoten »

josh2600 wrote: post #39
derefr wrote: post #14

Earlier quoted context omitted.

Sounds more like "greedy" than "optimistic" scheduling: the early jobs get the worms. In most scheduling systems, two 10,000-node jobs that each would saturate a cluster on their own will time-share if submitted together, with each job acting in practice more like 10,000 single-node jobs. The result is usually each job getting a probabilistic 50% share of the cluster while they're both running, and then whichever one…

This job swapping problem is a different sort of problem. In your example, I have cluster saturation as a concern. Google would love to saturate their cluster, but I would wager they usually are nowhere near saturation (scheduling a job on Borg takes something like 90 seconds at minimum; there's a lot of ground to cover). The innovation in optimistic schedulers is that you can have multiple schedulers running against…

Mesos is a two-level scheduler, where the elected master is more of a broker for offers from agents to external schedulers that decide to act on the offer or not. By default it's pessimistic, but this is pluggable and there's no reason you can't write an optimistic allocator module that will hand out offers to multiple schedulers.

Re: Nomad, a cluster manager and scheduler

#45

Post by josh2600 »

krenoten wrote: post #44
josh2600 wrote: post #39

Earlier quoted context omitted.

This job swapping problem is a different sort of problem. In your example, I have cluster saturation as a concern. Google would love to saturate their cluster, but I would wager they usually are nowhere near saturation (scheduling a job on Borg takes something like 90 seconds at minimum; there's a lot of ground to cover). The innovation in optimistic schedulers is that you can have multiple schedulers running against…

Mesos is a two-level scheduler, where the elected master is more of a broker for offers from agents to external schedulers that decide to act on the offer or not. By default it's pessimistic, but this is pluggable and there's no reason you can't write an optimistic allocator module that will hand out offers to multiple schedulers.

Part of the beauty of Omega is that there aren't just multiple schedulers, but schedulers with different profiles (this solves the head of line problem for long-running jobs while accelerating small jobs). Mesos, from the Borg paper[0], seems to have issues with head of line problems (including going so far as to silently fail jobs without queueing them).

FWIW, Omega seems like the answer if the overhead for resolving inconsistencies becomes low enough such that the benefits of optimistic scheduling outweigh the costs.

[0]http://www.slideshare.net/sameertiwari33/scheduling-on-large... See slide 6.

Re: Nomad, a cluster manager and scheduler

#46

Post by krenoten »

josh2600 wrote: post #45
krenoten wrote: post #44

Earlier quoted context omitted.

Mesos is a two-level scheduler, where the elected master is more of a broker for offers from agents to external schedulers that decide to act on the offer or not. By default it's pessimistic, but this is pluggable and there's no reason you can't write an optimistic allocator module that will hand out offers to multiple schedulers.

Part of the beauty of Omega is that there aren't just multiple schedulers, but schedulers with different profiles (this solves the head of line problem for long-running jobs while accelerating small jobs). Mesos, from the Borg paper[0], seems to have issues with head of line problems (including going so far as to silently fail jobs without queueing them). FWIW, Omega seems like the answer if the overhead for resolvin…

That link isn't working for me, unfortunately. I'm assuming that the HOL you're referring to stems from the assumption of the default DRF allocator of fast decisions, which can cause issues for mixed workloads if there's a lot of hoarding happening. This is something that could be addressed with an allocator module, but nobody has yet to my knowledge, leading me to think it's not a massive problem for many people. I think the people who care about this use case end up running multiple mesos clusters.

One note on priorities a la Borg (what I think people are actually using at Google today, but I don't know to what extent Omega was merged with it after the Omega paper or if Borg currently runs as an Omega scheduler or what), is that it sometimes causes annoyance for people at Google, as the optimal priority is not always assigned to certain workloads, leading to starvation or overpreemption of things that were not given a healthy number. I believe that Mesos' work on oversubscription took concerns around this issue into consideration.

It's really fascinating to compare the architecture of Nomad vs K8s, because both claim heavy descendence from Omega + Borg. If I squint, it seems like Nomad is more of a direct implementation based on those papers. K8s more explicitly modularizes more components, which I think is why a lot of companies are jumping behind it, as they can mold it in the ways they want.

Re: Nomad, a cluster manager and scheduler

#47

Post by fapjacks »

ninkendo wrote: post #41
fapjacks wrote: post #36

Earlier quoted context omitted.

I'm sure to a fan that "really freaking loves Mesos" you can't imagine other software competing with Mesos, but Hashicorp writes excellent software generally. Serf and Consul have completely sold me on Nomad, and I haven't even used it yet. I think there will be similar draw for many people that have used Hashicorp's software before.

It's just sad that there's yet another player that wants to be at the very bottom of the clustering stack. For physical resource management (memory and CPUs, etc), by definition you have to pick one solution and stick with it (because you don't want multiple cluster managers thinking they exclusively own the same hardware.) Mesos is the only player right now that's trying to do multitenancy; separating the concerns o…

Yeah, it feels almost like the mess of JavaScript libraries of the last five years: Everybody is dropping a hook into the water with some newfangled containerization technology, hoping to make it big.

Re: Nomad, a cluster manager and scheduler

#48

Post by Rapzid »

SEJeff wrote: post #37
fapjacks wrote: post #36

Earlier quoted context omitted.

I'm sure to a fan that "really freaking loves Mesos" you can't imagine other software competing with Mesos, but Hashicorp writes excellent software generally. Serf and Consul have completely sold me on Nomad, and I haven't even used it yet. I think there will be similar draw for many people that have used Hashicorp's software before.

No it isn't about koolaid, it is about technology. Consul, Vagrant, and Vault are all excellent Hashicorp technologies I've used. They are great, and some of the best in the industry for their problem spaces. You've got Mesos scaling to 10,000+ physical node clusters today in production at the likes of companies like Apple and Twitter. You've got Kubernetes being adopted and developed by pretty much all of the open s…

I'm sort of surprised to hear this view of the situation which is about the complete opposite of what I've had for the past few years. I have felt it's a shame that hashicorp have created consul, vault, a new raft implementation (which they got tons of flack for then etcd made THEIR OWN re-write which fixed a tons of issues this year..), serf, etc., and nobody has adopted or built on them. K8's current secrets solution is a bit underwhelming TBH, as an example.

Kubernetes has support for 250 node clusters in their list of blockers: https://github.com/kubernetes/kubernetes/blob/master/docs/ro...

Post reply on HN