Apache Mesos to be moved to Attic • Hacker News Board

Re: Apache Mesos to be moved to Attic

#101

Post by [deleted] » Tue, Apr 06, 2021, 11:58 PM UTC

aabhay wrote: post #67

random3 wrote: post #42

It's sad, but expected. This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark nor Apache Beam, HBase, etc. Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it and an elegant implemention that allowed for much more than what Kubernetes was desgined to. Kubernetes was supposed to be scheduler for Mesos and Google in…

100% Agreed. Ex-Mesosphere employee (I joined in 2015 and left/was fired a year later) The dominant ethos at Mesosphere was that they already won, and were poised to become the next 'cloud orchestration' above the cloud services. But the managers also had no empathy for developer experience -- the majority opinion was "distributed systems are hard, developers don't deserve to have a good experience", despite the new…

[deleted]

Re: Apache Mesos to be moved to Attic

#102

Post by yujie1984 » Wed, Apr 07, 2021, 12:02 AM UTC

benjamin_mahler wrote: post #71

I’m one of the long term PMC / committers on mesos. In retrospect I feel this was inevitable to a few key reasons: * k8s was a second system with all the learnings and experience of building such a system at Google for over a decade. Mesos was birthed by grad students and subsequently evolved into its position at Twitter but the engineers driving the project (myself included) did not have experience building cluster…

Well said Ben. I am also one of the long term PMC/Committer for the project.

One of the lesson I learnt was that Mesos's two level resource allocations was originally designed for running batch workloads (e.g., spark, mpi, etc.) if you look at the original paper. Use it to run long running services is actually an after thought. We end up finding that we have to do lots of tuning on the first level scheduling algorithm to ensure fairness given that the second level scheduler does not have the full view of the cluster and the first level scheduler does not have enough information to make good decisions. The solution to the problem is actually optimistic offer, which is essentially the k8s model.

Another reason k8s was successful is probably because the golang ecosystem. In mesos, we spent a lot of the energy building basic HTTP layer in C++ due to Mesos's unique threading model. I wish we could have spent those time working on actual useful features.

Re: Apache Mesos to be moved to Attic

#103

Post by chucky_z » Wed, Apr 07, 2021, 12:11 AM UTC

relistan wrote: post #48

The real gem of the Mesos ecosystem was the much lesser known Singularity scheduler from HubSpot. https://github.com/HubSpot/Singularity I've run this at scale, in production since 2015 and it has been absolutely rock solid and does most of the production things you'd want. Unlike commercial products, it was written to run HubSpot's own infrastructure so it does what a production system needs. Really bummed to have t…

There’s a Singularity driver for Nomad

https://www.nomadproject.io/docs/drivers/external/singularit...

Nomad is pretty darn easy to run and scales far. I help run about 20 clusters in day job from 100 to 1800 nodes, we’re going to push one of them to over 8500 nodes here soon too.

We don’t use the Singularity driver but we wrote two of our own and they work well and are easy to maintain.

Re: Apache Mesos to be moved to Attic

#104

Post by Conan_Kudo » Wed, Apr 07, 2021, 12:15 AM UTC

ksec wrote: post #25

I remember at MesosCon, Apple said they are the largest Mesos user. I wonder if they also moved to K8s as well.

Apple was at KubeCon last year, so I assume they've moved to Kubernetes too.

Re: Apache Mesos to be moved to Attic

#105

Post by thamer » Wed, Apr 07, 2021, 12:35 AM UTC

benjaminwootton wrote: post #23

I went to see Mesos early in their life in their San Francisco office after a joint customer put us in touch. Never in my life did I meet such an arrogant group of people. First, they left us waiting in reception for an hour. They eventually took the meeting over lunch, where we had to watch them inhaling their free food. Some guy in plastic-leather trousers spent most of the hour lecturing us about all of the multi…

Interesting to read this.

I interviewed 3 different engineers from Mesosphere in 2015 or so at a time when it seemed like the company was tanking and everyone was looking for a new job. At least for the people I interviewed, there was nothing to be arrogant about. We were specifically looking for engineers with this sort of experience and none of them were close to getting an offer.

Re: Apache Mesos to be moved to Attic

#106

Post by bogomipz » Wed, Apr 07, 2021, 12:37 AM UTC

yujie1984 wrote: post #102

benjamin_mahler wrote: post #71

I’m one of the long term PMC / committers on mesos. In retrospect I feel this was inevitable to a few key reasons: * k8s was a second system with all the learnings and experience of building such a system at Google for over a decade. Mesos was birthed by grad students and subsequently evolved into its position at Twitter but the engineers driving the project (myself included) did not have experience building cluster…

Well said Ben. I am also one of the long term PMC/Committer for the project. One of the lesson I learnt was that Mesos's two level resource allocations was originally designed for running batch workloads (e.g., spark, mpi, etc.) if you look at the original paper. Use it to run long running services is actually an after thought. We end up finding that we have to do lots of tuning on the first level scheduling algorith…

Thanks for the historical perspective. Might you or anyone else be able to recommend any resources that discuss the efforts to tune the two level scheduler for long-running workloads?

You mentioned: >"The solution to the problem is actually optimistic offer, which is essentially the k8s model."

Isn't the K8s model more "choose your QoS model" - BestError, Burstable or Guaranteed? Or am I misunderstanding your comment completely?

I was curious about the this:

>"Another reason k8s was successful is probably because the golang ecosystem. In mesos, we spent a lot of the energy building basic HTTP layer in C++ due to Mesos's unique threading model."

Could you say what was unique about the Mesos threading model?

Re: Apache Mesos to be moved to Attic

#107

Post by thamer » Wed, Apr 07, 2021, 12:38 AM UTC

dharmab wrote: post #31

jeffbee wrote: post #26

Earlier quoted context omitted.

Are there examples of high-utilization, large-scale Mesos deployments? Mesos didn't even gain over-commit until 2015, so it seems like it was generally behind the state of the art.

Most famously, Siri (used to?) run on a very large scale Mesos deployment (10000s of nodes, much higher than Kubernetes can scale to). Unfortunately the original article is lost, but here's a summary: https://daringfireball.net/linked/2015/04/29/siri-apache-mes...

Wayback Machine has it[1], but there's not much more content than in Gruber's summary.

[1] https://web.archive.org/web/20150429225603/https://mesospher...

Re: Apache Mesos to be moved to Attic

#108

Post by zekrioca » Wed, Apr 07, 2021, 12:46 AM UTC

ignoramous wrote: post #62

hintymad wrote: post #29

Earlier quoted context omitted.

A challenge with Mesos is that Mesos was a piece of technology, a framework at most, instead of a product. When I was using Mesos, the selling point was flexible and efficient resource scheduling. Unfortunately, resource/machine efficiency alone does not sell well, as most of the companies and individuals have betters things to worry about, say, productivity.

> Unfortunately, resource/machine efficiency alone does not sell well... Surprising because one of the driving forces behind accelerating adoption of on-demand IaaS and various PaaS like Serverless is that too many expensive server resources lay idle. According to James Hamilton, chief Data Center architect at AWS, server utilisation remains very low (10%–15%) despite servers being the most dominant cost of building…

One really does not want to saturate a fleet of servers to a point where random faults and broken SLOs are popping up all over the place. That’s first and foremost the main reason datacenter utilization is low.

Edit: sentence fixing

Re: Apache Mesos to be moved to Attic

#109

Post by jqcoffey » Wed, Apr 07, 2021, 12:48 AM UTC

jeffbee wrote: post #26

kraemate wrote: post #16

Mesos made a ton of new contributions to distributed resource management. Resource offers and application integration can allow high cluster utilization and efficiency. Giving applications the opportunity to take part in resource allocation and scheduling was similar to the Exokernel design, and led to many interesting middleware architectures. Mesos also introduced Dominant Resource Fairness for allocating resources…

Are there examples of high-utilization, large-scale Mesos deployments? Mesos didn't even gain over-commit until 2015, so it seems like it was generally behind the state of the art.

There must be some folks from Criteo lurking here. I'm an ex-Criteo'er and if memory serves we had something on the order of 10K nodes running mesos/marathon. We did all kinds of silly things to it, like running very CPU intensive .NET/Core apps.

I dug this post up showing a service performing an internal auction of up to 530M advertising campaigns/sec on 88K CPUs in Mesos: https://medium.com/criteo-engineering/migrating-arbitrage-to...

We also ran HiveServer2 and the Hive Metastore in Mesos, though that wasn't super CPU intensive (that was a pain, but mostly due to our Kerberos deployment).

The general use case of Mesos/Marathon always worked for us just fine (self-executable JVM apps), though there was plenty of Mesos hate at Criteo (and eventually Kubernetes spun up, though I left about a year ago and don't know its footprint).

PS, Hi Greg S! Hi Maxime B! <-- if you're reading :).

Re: Apache Mesos to be moved to Attic

#110

Post by jaaron » Wed, Apr 07, 2021, 1:15 AM UTC

meatmanek wrote: post #100

jaaron wrote: post #39

I preferred Mesos to k8s. I think it's core architecture (a 2-level scheduler) is a better foundation. For the longest time, I felt k8s was effectively an overgrown hobby project that had no place being deployed the way it was. That had me realize something in the shower this morning: k8s is the Rails of the cloud. Back when Rails came out, it too was a bit of a hobby project. When coming from more established enterp…

I actually think Kubernetes is a better foundation for a 2-level scheduler system than Mesos is. (In k8s land, they call this the operator pattern[1]). Each operator creates Pod objects in k8s with constraints/affinity/anti-affinity, and the K8s scheduler decides on your operator's behalf where each pod will go. Pending pods (pods that aren't assigned to boxes yet) are also a really useful signal for cluster autoscal…

My frame of reference for the argument about Mesos vs k8s is from earlier in the timeline when the k8s was up and coming and just starting to compete with Mesos. Just as in my analogy with Rails: at it's inception, it lacked a bunch of features, but it was accessible. Years later, Rails had many of those features.

k8s is the same. Operators didn't come around until late 2016 by CoreOS and even then they weren't widely adopted. It wasn't until after RedHat bought CoreOS and pushed operators as a pattern into k8s that the pattern took off. As it is, the k8s version of the operator framework only went 1.0 last year.

Finally, I should have pointed it out in my original comment, the Mesos vs k8s comparison isn't perfect apples and apples. Mesos is just a component in a stack, whereas k8s is effectively a full stack. Again, Rails was pitched as an opinionated, batteries-included framework compared to many of the more focused frameworks that, for their niche, may have been better, but the convenience of having everything in one packages won out.