Docker vs. Kubernetes vs. Mesos

Of cuz, Google open sourced a container runtime before in c++, and no one cared.

Why should there be an assumption of (full) objectivity on an article written by one provider of the 3 technologies that are being compared? Whit that said the article is fairly balanced in presenting some layers of the whole context (mostly the historical one).

Although I would mention Google tried to have their own Docker but that didn't pan out (https://github.com/google/lmctfy) so they switched to Docker and had Kubernetes open-sourced. More historical details would have just made for a longer (albeit more interesting) article I guess.

Now to actual context. While they mention Swarm is not in CNCF and under tight control of Docker Inc. they don't mention that while Mesos is in ASF, Mesosphere hired the majority of the PMC (committers with voting rights). and the rest of the DC/OS is not even in ASF so doesn't even have to abide to the rules of the ASF. At the same time, Mesosphere heavily used the Mesos brand by mixing Mesos-phere and DC/OS into everything Mesos.

So the reason people talk about Mesos, Marathon, DC/OS and Mesosphere as almost synonyms is because they made it so. Marathon used to be a Mesos framework and service scheduler, now it's a DC/OS one for the most part (https://news.ycombinator.com/item?id=13656193)

However, in the process, they managed to alienate a part of the community too, all this while Kubernetes was able to do probably one of the best community jobs in OSS and skyrocketed (https://trends.google.com/trends/explore?q=mesos,kubernetes). That must be a bitter irony if you consider that initially Kubernetes was supposed to be just a Mesos framework...

So yes, Mesos is lower level and with the two phase scheduling it should be more versatile, etc. but that value is highly diminished if you consider the focus is around DC/OS.

I think there are several angles that make this whole context interesting. Perhaps it would be worth a full writing...

Uh. I'm skeptic about Swarm being something that "works today". I really want to use it and spent some time experimenting, but failed to get it working. It could work for a toy webapp project, but I believe is insufficient for anything complicated. There are a lot of things that one would expect to have but that are yet unsolved. Or I'm just unaware that there is a solution (or the solution didn't fit my personal requirements).

1. Networking options are really limited.

Built-in ingress loses of the originating IP address, cannot bind ports only on specific nodes (something that may have partially fixed this was implemented in 17.06 with --network=host option, but I haven't found much documentation), cannot prefer same-node containers (which is important if you have many microservices, as the latency adds up quickly) and more.

For ingress, if one's lucky to have only HTTP traffic (I don't), they can use something like Traefik. But they'll need to run LBs on a manager nodes which isn't really a smart thing to do, as Swarm is said to be sensitive to overloaded managers. Or one can just go for external LBs, like CloudFlare or whatever Amazon/Google/Microsoft offers.

If one needs raw TCP, UDP or other IP traffic - I think they'd better completely ignore built-in service discovery and LBs and go for something external, like etcd+haproxy/nginx, outside of the Swarm (on nodes' host OS). It has to be completely manual setup (okay, I meant Ansible/Puppet/Chef/Salt/etc). While it's possible to run LBs with Docker (non-Swarm containers, if the Swarm network is attachable), etcd is just not designed to auto-deploy on Swarm: its design explicitly disallows one to just `docker service create --name etcd my/custom/etcd && docker service scale etcd=5`, you'll need to set up every node by hand. I believe Consul is better in this regard but haven't tried it yet.

2. I haven't figured out how to have a persistent storage that follows the containers. If a node dies, Swarm would spawn new containers on other nodes, but no chance to have even a slightly dated snapshot. There was something called Flocker that looked like a solution, but it's essentially dead (despite the revival attempts). GlusterFS is an option I know about, but it's really sensitive to latency.

Databases are even more tricky, unless one's bold to use something fancy like CockroachDB (I had enough subtle issues with RethinkDB to be wary about bleeding edge stuff). Maybe I'm just too stupid, but I failed to grok dynamic PostgreSQL multi-master BDR setup, so my DB is still SPOF with some WAL streaming replication manually-activated failovers.

3. Secrets look like a nice addition, but they're best avoided. They're immutable and you have to recreate the container to switch them. If you have any services that have many user-initiated long-living connections (e.g. IRC, XMPP, Websockets or media streaming), this would makes secrets basically unusable for anything that could be rotated, like TLS certificates. Unless you can drop all your users every now and then.

4. Logging was quite messy, but they've sorted it out with 17.06.

(As for the K8S - it solves most of the issues, but I got my share of issues with Rancher, so I'm really wary about having any complexity in the core. There's already a beast called Linux kernel down there, and $deity have mercy on those who have to debug its oopses. If a behemoth - I mean, K8s - decides to misbehave, I expect to have a really bad time trying to keep things afloat. Even Swarm mode is fairly complex black box binary - but at least I can try debugging it.)

> initially Kubernetes was supposed to be just a Mesos framework...

This is not correct. There is a community-owned project that allows Kubernetes to run on Mesos, but it came after standalone Kubernetes.

Disclosure: I work on Kubernetes at Google.

Then perhaps you could ask John Wilkes ;)

later edit: I'm referring to the fact that when Kubernetes came out it didn't have resource allocation and the answer to "how it compares to Mesos" was that it would run on top of Mesos as a framework.

John Wilkes gave a talk about Kubernetes at MesosCon in 2014 https://www.youtube.com/watch?v=VQAAkO5B5Hg and also answered the above question quite a bit while there...