Can someone explain the difference between Apache Mesos, Apache Spark, and Kubernetes. As someone reading just announcements, and having never experimented with any of these tools, they sound like they promise the same capabilities. Is it just that they are different open source projects aiming for the same goal ... or are their goals different?
I'll give it a shot, with the disclaimer that this explanation is based on conference talks I've seen and documentation I've read. Spark is a successor to Hadoop, aiming to perform big distributed data crunching jobs more quickly by not limiting itself to a map-reduce paradigm and by holding more data in memory. Apache Mesos is a resource scheduler for a cluster of machines. It is architected to be agnostic about the…
Mesosphere Marathon, which sits on top of Mesos, makes doing long-running services easier; it starts services, restarts them if machines crash, etc etc. Where Mesos is the resource schedular, Marathon is more like a *nix init system.
Aurora is supposed to do much the same thing, but is in Incubation at the moment. There's also "Singularity", which does a bunch of stuff including managing long-running services and one-off services through an HTTP API and webapp.
Mesos is supposed to be treated sort of as a "cluster kernel", with "frameworks" which sit on top of it and use it to schedule things. It's a lot more versatile than "I want to deploy a bunch of services" - it's more along the lines of having your own EC2 that you can request resources from on-demand.