Apache Mesos to be moved to Attic • Hacker News Board

Re: Apache Mesos to be moved to Attic

#91

Post by p_l » Tue, Apr 06, 2021, 9:52 PM UTC

orthecreedence wrote: post #47

dragonsh wrote: post #7

Earlier quoted context omitted.

For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. A good alternative to kubernetes is LXD [1] or just stick with docker compose. Kubernetes except for managed services from cloud providers is more difficult than an average application to manage and a huge cost in itself to run and maintain. [1] https://www.linuxcontainers.org/

I don't know about this. I caution people against microservices architecture all the time, but at my company having a scheduler just made sense. We spin up and down queue workers by massive amounts every hour, and doing this with anything besides a scheduler would be really tricky. Granted, we use Nomad, not k8s, but we definitely need a scheduler and definitely are not reaching Google-scale.

It doesn't even have to be some microservices monster, just trying to bin-pack a bunch of different services, even monoliths, onto a fleet of servers.

If I did it how the usual "k8s is only for FAANG scale" people tell me to do, I'd go bankrupt ;)

Re: Apache Mesos to be moved to Attic

#92

Post by ForHackernews » Tue, Apr 06, 2021, 10:02 PM UTC

orf wrote: post #90

ForHackernews wrote: post #86

Earlier quoted context omitted.

From the docs you linked: > ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType. Internal only. > NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Se…

> This one's a new one to me, and apparently relies on some special new widgets. ExternalName has been around since 2016. > Nobody uses NodePort to expose external services directly, and I think you know that. Sure they do. Anyone using a LoadBalancer does this implicitly. If you don’t want k8s to manage the allocated port or want to use something bespoke that k8s doesn’t offer out of the box then using a NodePort is…

My criticism of k8s is it's an absolutely batshit level of complexity[0] that somehow still fails to provide extremely basic functionality out-of-the-box (unless paired with a whole cloud provider ecosystem, but then why not just skip k8s and use ECS???). I don't think k8s solves real problems most developers face, but it does keep all these folks[1] getting paid, so I can see why they'd advocate for it.

Nomad is vastly superior in every way except for mindshare; Elastic Beanstalk or Dokku is superior for most normal-people use cases.

[0] https://kubernetes.io/docs/concepts/overview/components/

[1] https://landscape.cncf.io/

Re: Apache Mesos to be moved to Attic

#93

Post by qbasic_forever » Tue, Apr 06, 2021, 10:11 PM UTC

waynesonfire wrote: post #79

qbasic_forever wrote: post #76

Earlier quoted context omitted.

It's refreshing to see an honest and critical evaluation of things. This kind of decision should be celebrated and encouraged with all projects.

On the announcement that it's getting shelved you get a retro. How is that refreshing? Whats your baseline? Where is D2iQ response to this news?

Refreshing in the sense that it's a look back at what worked and what maybe didn't work so other projects don't make the same mistake. They could have just flipped the code repo to "archived" and moved on without a word.

Re: Apache Mesos to be moved to Attic

#94

Post by kraemate » Tue, Apr 06, 2021, 10:16 PM UTC

jeffbee wrote: post #26

kraemate wrote: post #16

Mesos made a ton of new contributions to distributed resource management. Resource offers and application integration can allow high cluster utilization and efficiency. Giving applications the opportunity to take part in resource allocation and scheduling was similar to the Exokernel design, and led to many interesting middleware architectures. Mesos also introduced Dominant Resource Fairness for allocating resources…

Are there examples of high-utilization, large-scale Mesos deployments? Mesos didn't even gain over-commit until 2015, so it seems like it was generally behind the state of the art.

IIRC you could always overcommit in Mesos using DRF weights and accepting resource offers in your application. I could be wrong.

The larger point is that Mesos introduced a new, exciting way to do truly distributed allocation (where the cluster manager (i.e., Mesos) and various applications coordinated and cooperated in how they use computing resources). In contrast, Kubernetes is centralized, pretty vanilla, and I would love to know what new ideas it has introduced (from an algorithmic and architecture perspective).

Re: Apache Mesos to be moved to Attic

#95

Post by aabhay » Tue, Apr 06, 2021, 10:23 PM UTC

dharmab wrote: post #72

aabhay wrote: post #67

Earlier quoted context omitted.

100% Agreed. Ex-Mesosphere employee (I joined in 2015 and left/was fired a year later) The dominant ethos at Mesosphere was that they already won, and were poised to become the next 'cloud orchestration' above the cloud services. But the managers also had no empathy for developer experience -- the majority opinion was "distributed systems are hard, developers don't deserve to have a good experience", despite the new…

Ex-DC/OS user. This explains a lot. One thing I'd like to say is that CoreOS was the best component of DC/OS by a long mile. We still use its fork (Flatcar) today.

That’s cool! The issue is that when you’re selling to a massive legacy institution, the infosec on a new Linux distro that self updates to new patches is really no bueno. We lost many months on this piece and on key deals. I heard from the grapevine that they ended up having to port DCOS to Ubuntu after all, soon after I left.

Re: Apache Mesos to be moved to Attic

#96

Post by jaaron » Tue, Apr 06, 2021, 11:03 PM UTC

star-trek-fleet wrote: post #57

random3 wrote: post #42

It's sad, but expected. This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark nor Apache Beam, HBase, etc. Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it and an elegant implemention that allowed for much more than what Kubernetes was desgined to. Kubernetes was supposed to be scheduler for Mesos and Google in…

> Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it Brilliant on paper. The idea was influenced by MR workloads on Borg. Because MR is so large a workload pattern on Borg, that it benefits from having its own scheduler, and Borg also benefit from unnecessary meddling. The idea was only brilliant for a very small number of human users, both inside Google and in the broader industry. In terms…

>> Kubernetes was supposed to be scheduler for Mesos and Google invested in it.

> I never heard of such plan.

Oh, there was a plan:

https://github.com/kubernetes-retired/kube-mesos-framework

But it wasn't really in anyone's business model interests, even if it may have made sense for customers and the community.

Re: Apache Mesos to be moved to Attic

#97

Post by _jal » Tue, Apr 06, 2021, 11:12 PM UTC

benjaminwootton wrote: post #23

I went to see Mesos early in their life in their San Francisco office after a joint customer put us in touch. Never in my life did I meet such an arrogant group of people. First, they left us waiting in reception for an hour. They eventually took the meeting over lunch, where we had to watch them inhaling their free food. Some guy in plastic-leather trousers spent most of the hour lecturing us about all of the multi…

> I had a similar experience with other west coast tech companies.

I am the infra lead for a "west coast tech company", and they gave us the same guff. I knew we would not be using them after the first few minutes of listening to them.

But please do continue to hate on this coast, tell your friends. I don't want to compete with more folks for housing.

Re: Apache Mesos to be moved to Attic

#98

Post by justicezyx » Tue, Apr 06, 2021, 11:26 PM UTC

jaaron wrote: post #96

star-trek-fleet wrote: post #57

Earlier quoted context omitted.

> Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it Brilliant on paper. The idea was influenced by MR workloads on Borg. Because MR is so large a workload pattern on Borg, that it benefits from having its own scheduler, and Borg also benefit from unnecessary meddling. The idea was only brilliant for a very small number of human users, both inside Google and in the broader industry. In terms…

>> Kubernetes was supposed to be scheduler for Mesos and Google invested in it. > I never heard of such plan. Oh, there was a plan: https://github.com/kubernetes-retired/kube-mesos-framework But it wasn't really in anyone's business model interests, even if it may have made sense for customers and the community.

Sure, this type of early idea happens to any high-profile project.

I meant to say that that I never heard any significant decision maker had that plan on K8s' future.

Re: Apache Mesos to be moved to Attic

#99

Post by dharmab » Tue, Apr 06, 2021, 11:39 PM UTC

aabhay wrote: post #95

dharmab wrote: post #72

Earlier quoted context omitted.

Ex-DC/OS user. This explains a lot. One thing I'd like to say is that CoreOS was the best component of DC/OS by a long mile. We still use its fork (Flatcar) today.

That’s cool! The issue is that when you’re selling to a massive legacy institution, the infosec on a new Linux distro that self updates to new patches is really no bueno. We lost many months on this piece and on key deals. I heard from the grapevine that they ended up having to port DCOS to Ubuntu after all, soon after I left.

Yeah we disabled the auto updates and orchestrated the updates ourselves. Saved our bacon multiple times when kernel/systemd/docker bugs got pushed out.

Re: Apache Mesos to be moved to Attic

#100

Post by meatmanek » Tue, Apr 06, 2021, 11:40 PM UTC

jaaron wrote: post #39

I preferred Mesos to k8s. I think it's core architecture (a 2-level scheduler) is a better foundation. For the longest time, I felt k8s was effectively an overgrown hobby project that had no place being deployed the way it was. That had me realize something in the shower this morning: k8s is the Rails of the cloud. Back when Rails came out, it too was a bit of a hobby project. When coming from more established enterp…

I actually think Kubernetes is a better foundation for a 2-level scheduler system than Mesos is. (In k8s land, they call this the operator pattern[1]). Each operator creates Pod objects in k8s with constraints/affinity/anti-affinity, and the K8s scheduler decides on your operator's behalf where each pod will go.

Pending pods (pods that aren't assigned to boxes yet) are also a really useful signal for cluster autoscaling that is annoying to calculate in Mesos. (The Mesos master has no idea how many pods each framework _wants_ to launch, so we ended up writing code that knows how to deduce the resource demand from each framework's API or UI.)

On the other side of the same coin, the framework in Mesos has no idea the total resources available to the cluster - it has to wait to be offered each box's resources in turn. This usually means frameworks either:

a) accept the first offer which matches the basic requirements for a task, even if there are better places to run that task, or b) accept every single offer, under the assumption that they're the only framework on the cluster, and then implement their own scheduler on top of this pool of resources. (I call this the "monoframework" approach.)

The former approach is how Mesos is supposed to work, but can lead to all sorts of bad outcomes, e.g. all the copies of a Marathon service running on the same box, or Spark having to use timeouts to know when it should give up waiting for offers if it doesn't get enough right away.

The latter approach can lead to better placements, but defeats the purpose of the two-level scheduler, as no other frameworks can use the resources that aren't being used by the monoframework.

Under Kubernetes, any operator can query the state of the cluster and make informed decisions about what to request.

1. https://kubernetes.io/docs/concepts/extend-kubernetes/operat...