Apache Mesos to be moved to Attic • Hacker News Board

Re: Apache Mesos to be moved to Attic

#61

Post by xyzzy_plugh » Tue, Apr 06, 2021, 6:53 PM UTC

nemothekid wrote: post #46

xyzzy_plugh wrote: post #6

I used Mesos for a few years before experiencing Kubernetes. As neat as Mesos was, it was doomed from the start. For one, Kubernetes was, at least to some extent, a rewrite and extraction of functionality built at Google, from their production orchestration ecosystem that is Borg. The fact that Kubernetes was heavily influenced by a successful, large solution in this space allowed it to leapfrog, at least a bit, the…

> As neat as Mesos was, it was doomed from the start. I don't think it was doomed from the start; Mesos was deployed at Twitter 4 years before k8s was even released. The "problem" with Mesos was that it was focused on a future that never panned out. The "killer app" for Mesos was Spark, and the problem they were focused on solving was resource allocation for batch jobs. Mesos was supposed to be a replacement for YARN…

You're partially correct.

Mesos was originally an academic project out of UC Berkeley. I'm not aware of what industry connections there were, but the story is not at all similar to Kubernetes and Borg.

Aurora was Twitter's contribution -- a bit missing piece of Mesos at the time. It definitely steered Mesos towards solving for Spark, but I'm really not sure what Mesos was actively solving before then.

Re: Apache Mesos to be moved to Attic

#62

Post by ignoramous » Tue, Apr 06, 2021, 7:09 PM UTC

hintymad wrote: post #29

xyzzy_plugh wrote: post #6

I used Mesos for a few years before experiencing Kubernetes. As neat as Mesos was, it was doomed from the start. For one, Kubernetes was, at least to some extent, a rewrite and extraction of functionality built at Google, from their production orchestration ecosystem that is Borg. The fact that Kubernetes was heavily influenced by a successful, large solution in this space allowed it to leapfrog, at least a bit, the…

A challenge with Mesos is that Mesos was a piece of technology, a framework at most, instead of a product. When I was using Mesos, the selling point was flexible and efficient resource scheduling. Unfortunately, resource/machine efficiency alone does not sell well, as most of the companies and individuals have betters things to worry about, say, productivity.

> Unfortunately, resource/machine efficiency alone does not sell well...

Surprising because one of the driving forces behind accelerating adoption of on-demand IaaS and various PaaS like Serverless is that too many expensive server resources lay idle. According to James Hamilton, chief Data Center architect at AWS, server utilisation remains very low (10%–15%) despite servers being the most dominant cost of building and running a data center (which is to say, folks pay through their nose for servers yet those are under-utilized by a huge margin) [0]

[0] https://youtu.be/dInADzgCI-s?t=535

Re: Apache Mesos to be moved to Attic

#63

Post by hintymad » Tue, Apr 06, 2021, 7:12 PM UTC

ignoramous wrote: post #62

hintymad wrote: post #29

Earlier quoted context omitted.

A challenge with Mesos is that Mesos was a piece of technology, a framework at most, instead of a product. When I was using Mesos, the selling point was flexible and efficient resource scheduling. Unfortunately, resource/machine efficiency alone does not sell well, as most of the companies and individuals have betters things to worry about, say, productivity.

> Unfortunately, resource/machine efficiency alone does not sell well... Surprising because one of the driving forces behind accelerating adoption of on-demand IaaS and various PaaS like Serverless is that too many expensive server resources lay idle. According to James Hamilton, chief Data Center architect at AWS, server utilisation remains very low (10%–15%) despite servers being the most dominant cost of building…

I don't deny that. It's just that so many companies have so much inefficiency else where that addressing resource inefficiency has too low a marginal return.

Re: Apache Mesos to be moved to Attic

#64

Post by bdd » Tue, Apr 06, 2021, 7:20 PM UTC

jeffbee wrote: post #55

bdd wrote: post #43

Earlier quoted context omitted.

Twitter. From generic caches to ad serving; from stream processing to video encoding, all high utilization applications of either one or multiple schedulable resources.

As MesosCon Twitter said their cluster utilization was between 20 and 30%.

These jobs had their allotted quotas, per team, giving them above >70% utilization in their logical slice of the cluster. E.g. video processing team gets 20,000 nodes globally. They stack (co-locate) their tasks (interpret: set of processes) however they want.

Granted Twitter operated one big shared Mesos+Aurora offering for everything*, the whole cluster high utilization wouldn't give much flexibility to absorb load, or do reasonable capacity planning (which was an entire org in itself) when you own and operate those machines and data centers. I can't comment much on the 20-30% figure given in MesosCon, it's been more than 5 years since I was last privy to these figures.

Re: Apache Mesos to be moved to Attic

#65

Post by tknaup » Tue, Apr 06, 2021, 7:23 PM UTC

nemothekid wrote: post #46

xyzzy_plugh wrote: post #6

I used Mesos for a few years before experiencing Kubernetes. As neat as Mesos was, it was doomed from the start. For one, Kubernetes was, at least to some extent, a rewrite and extraction of functionality built at Google, from their production orchestration ecosystem that is Borg. The fact that Kubernetes was heavily influenced by a successful, large solution in this space allowed it to leapfrog, at least a bit, the…

> As neat as Mesos was, it was doomed from the start. I don't think it was doomed from the start; Mesos was deployed at Twitter 4 years before k8s was even released. The "problem" with Mesos was that it was focused on a future that never panned out. The "killer app" for Mesos was Spark, and the problem they were focused on solving was resource allocation for batch jobs. Mesos was supposed to be a replacement for YARN…

Co-founder of Mesosphere/D2iQ here.

Mesos started as a research project at Berkeley in 2009 and was originally focused on cluster computing frameworks like Hadoop. From the paper: "We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI." It actually predates YARN by a few years. But, it very quickly (in 2010) saw production use at Twitter as the foundation for Twitter's custom PaaS which was later open sourced as Apache Aurora.

Marathon's main use case was actually for running microservice application in containers, which is why it has some advanced features around managing groups of containerized apps and their dependencies. The "meta-framework" use case for launching custom frameworks was also important but basically just needs Marathon to keep a container alive. Mesosphere never made Marathon proprietary. The full code is still OSS here: https://github.com/mesosphere/marathon/ Our commercial product DC/OS just added advanced workflows through a UI on top, and better integration with the rest of the components around Mesos.

Re: Apache Mesos to be moved to Attic

#66

Post by marcinzm » Tue, Apr 06, 2021, 7:43 PM UTC

ForHackernews wrote: post #60

marcinzm wrote: post #51

Earlier quoted context omitted.

Your points mostly only matter if you're running on bare metal. If you're in the cloud then you've got load balancers and databases covered by your cloud provider. I need Kubernetes to handle problems that I already have great solutions for. I want it to handle the problems that my cloud provider provides poor or very specialized (ie: lock in) solutions for. Which for me it does very well and a lot more easily than d…

ITT: K8s is great because it frees us from the tyranny of Big Cloud providers. Also ITT: Oh, but of course k8s is totally unusable unless you buy it from a Big Cloud provider.

Please don't put words in my mouth. I said it frees you from lock in by a single cloud provider which it does.

Re: Apache Mesos to be moved to Attic

#67

Post by aabhay » Tue, Apr 06, 2021, 7:45 PM UTC

random3 wrote: post #42

It's sad, but expected. This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark nor Apache Beam, HBase, etc. Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it and an elegant implemention that allowed for much more than what Kubernetes was desgined to. Kubernetes was supposed to be scheduler for Mesos and Google in…

100% Agreed. Ex-Mesosphere employee (I joined in 2015 and left/was fired a year later)

The dominant ethos at Mesosphere was that they already won, and were poised to become the next 'cloud orchestration' above the cloud services. But the managers also had no empathy for developer experience -- the majority opinion was "distributed systems are hard, developers don't deserve to have a good experience", despite the new cool easy-to-use distributed systems cropping up every week. The company even developed a sham 'community edition' that was designed to fail. One person complained on our community slack channel that he left his cluster running overnight and was charged $600 for the 12 hours of use.

Within a few months of joining, I started to point out their odd technical decisions (e.g. why they decided to build their enterprise edition on Core OS rather than a trustworthy distro), and was eventually chopped for speaking up. I was fired right before a massive company offsite. When the rest of my team came back, they along with my manager were fired, too. The messsage was: we didn't need your whole team but because you spoke up we had to double screw you.

Re: Apache Mesos to be moved to Attic

#68

Post by yjftsjthsd-h » Tue, Apr 06, 2021, 7:50 PM UTC

fastball wrote: post #15

fiberoptick wrote: post #2

End of an era... Are there any viable alternatives to Kubernetes and Nomad?

Docker-swarm?

Having worked at a company running Docker swarm at... medium(?) scale... I have witnessed a truly shocking variety of bugs in its network stack. I always wondered if it was something specific to the company setup, but the result is that we just couldn't stay on a platform that would randomly fail to give containers a working network connection to each other.

Re: Apache Mesos to be moved to Attic

#69

Post by streblo » Tue, Apr 06, 2021, 7:51 PM UTC

jeffbee wrote: post #55

bdd wrote: post #43

Earlier quoted context omitted.

Twitter. From generic caches to ad serving; from stream processing to video encoding, all high utilization applications of either one or multiple schedulable resources.

As MesosCon Twitter said their cluster utilization was between 20 and 30%.

I worked for Twitter up until 2017 and when I was there it was much higher than 20-30%, definitely >50%. It's very possibly changed since then, but at least at that point in time Twitter was running Mesos on many thousands of machines.

Re: Apache Mesos to be moved to Attic

#70

Post by aabhay » Tue, Apr 06, 2021, 7:55 PM UTC

jaaron wrote: post #45

random3 wrote: post #42

It's sad, but expected. This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark nor Apache Beam, HBase, etc. Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it and an elegant implemention that allowed for much more than what Kubernetes was desgined to. Kubernetes was supposed to be scheduler for Mesos and Google in…

This is much more the real story but I doubt most folks will ever hear it. Totally agree. Mesos project always had a trouble with governance and didn't build out the larger community the way other projects did. If they had built that coalition, made the project more accessible to others, who knows, maybe it would have gone differently. Then again, k8s succeeded in part due to the Google reputation (even if undeserved…

I think the problem went deeper than that actually. Mesos required that applications use a distributed system API to interface with the executors. This meant it was very challenging to write a distributed system upon Mesos, even though that was ostensibly the very thing that Mesos was meant to solve. When K8s came along and made everything manifest and YAML-based, we sort of knew that we were screwed because of how much simpler that was.