How much of this could be avoided if the application didn't use mongo? Needing to run a three-nice cluster out of the gate seems like a big part of the problem. Sure, you want backups and redundancy for any database, but there are situations where a MySQL or pg slave that can be switched on makes more sense financially, especially if load doesn't require a three-node cluster.
I Want to Run Stateful Containers, Too
11–20 of 102 posts
Re: I Want to Run Stateful Containers, Too
#12The author is correct. We should be running stateful containers, but until the the software is ready the default advice quite rightly should be not to unless you understand the implications. For example we run ceph and use it in kubernetes. We obviously only run replica sets (the kubernetes feature, for those uninitiated) of 1 for these services. There's also some rough edges with locking for dead nodes, but it's def…
Does kubernetes not have a mechanism for defining host/rack diversity constraints?
Re: I Want to Run Stateful Containers, Too
#13I think the author is implicitly describing the paradigm shift from software and hardware abstractions to service abstractions. We used to make computers by soldering electronics together, then by assembling cases and components, then buying premade servers, then renting cloud capacity, and now we're starting to rent everything as services. The cloud is about going from capex to opex and the "capex" now is the initia…
How does OpenStack stack up in this regard? A step in the right direction? https://en.wikipedia.org/wiki/OpenStack
Terrible usability, and mainly aiming at the lower levels of the stack, not the level of actually deploying applications. It is far easier to get a container stack running than to bother with Open Stack.
Re: I Want to Run Stateful Containers, Too
#14We run a large stateful service with hundreds of images based on whaleware and a thin layer of orchestration on top, and it works. We just dont expect everything to happen magically through an orchestration tool.
Re: I Want to Run Stateful Containers, Too
#15The author is correct. We should be running stateful containers, but until the the software is ready the default advice quite rightly should be not to unless you understand the implications. For example we run ceph and use it in kubernetes. We obviously only run replica sets (the kubernetes feature, for those uninitiated) of 1 for these services. There's also some rough edges with locking for dead nodes, but it's def…
There are still a whole lots of pieces of the puzzle missing.
I think it's all about creating ready to use containers which are running on different hosts which can replicate their data between themselves and handle failovers (they need to connect to etcd or zookeeper or consul for that).
For PostgreSQL it seem to have already solved (?) 2 times by different people:
https://github.com/zalando/patroni https://flynn.io/
But we don't only need it for PostgreSQL.
You need ready to use data-replicating Docker images for storing/indexing log files.
And for statistics.
Probably ready to use containers for replicated/clustered Redis.
We need something for a replicated S3-like service for storing static files.
Which can be used for different things like: - let's say you are running a website with Wordpress it can be configured to put it's files in a 'CDN'. So now whenever you you deploy Wordpress you point reverse proxy with caching (like Varnish) at your S3-like service.
A S3-like service could also be used to hold your own Docker images. You could put a Docker registry in front of it to push and pull to.
We need something, maybe also S3-type service, for storing the files of your local git repositories (which holds the sources of your Docker-images).
As far as I know in the ecosystem there is still a scheduler missing which can deploy these containers on the right hosts.
And there is also no standard API for starting new containers or whole machines when the other containers (or some monitoring tool) in a cluster noticed one is missing.
These things take time. Lots of time. :-(
Re: I Want to Run Stateful Containers, Too
#16Not sure why the author complaining about VM & cloud ?
You can get real machines through APIs not from Amazon, but you can from other providers like: Rackspace and IBM/Softlayer (and others). He even links to Bryan Cantrill, so pretty sure Joyent can deliver containers (even Docker ?) on baremetal if you want them.
Re: I Want to Run Stateful Containers, Too
#17I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own. But this is the bit that surprises me: They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. $1500 per month, versus weeks of engineer…
Yes, I think devs sometimes forget that this is the bleeding-est of bleeding edges. Hell, I can't even get a normal Digital Ocean server instance to programmatically drop and recreate doing minimal things like creating a user and setting SSH keys. I suppose if I wanted to learn something heavyweight like Chef or Puppet I could, but for now, I just maintain a list of commands to run, and understand that Digital Ocean servers can't quite be treated like livestock just yet, though I can fake it pretty well. Eventually Digital Ocean will get around to making the platform changes, which are all already in feature request.
Engineering time is very limited and the support cases are exponentially large. The author wanted MongoDB supported in the hard case. Well, that means you also need to support 4 or 5 other, similar databases the same way.
If you want a bleeding-edge PaaS with corporate support, you should expect to pay corporate prices. If the project / organization doesn't allow for that, then either the requirements or the mindset is off-kilter.
I think the what the author is really mad about is the vendor lock-in. That used to bother me too, until I realized that learning new platforms is done on my employer's dime and not mine.
A few more years, and some other challenge will be on the bleeding edge and this one will be mostly solved, meaning that open-source community solutions will exist that you can build once and forget about it. Knowing most devs, they'll spend all their time complaining about whatever the new edge is instead of marveling at how much easier things are now compared to the 'bad old days' of 2015.
Re: I Want to Run Stateful Containers, Too
#18I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own. But this is the bit that surprises me: They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. $1500 per month, versus weeks of engineer…
> And so it is with PaaSes. The marker was passed years ago. We don't need to go on these spiritual quests any more.
The CloudFoundry installation instructions alone makes me run away screaming. Same with OpenStack.
There's no "spiritual quest" involved, but a lack of willingness to add unnecessary complexity. Of course it looks different when you've already eaten the cost of getting it up and running and/or has readily available in-house expertise.
Re: I Want to Run Stateful Containers, Too
#19The author is correct. We should be running stateful containers, but until the the software is ready the default advice quite rightly should be not to unless you understand the implications. For example we run ceph and use it in kubernetes. We obviously only run replica sets (the kubernetes feature, for those uninitiated) of 1 for these services. There's also some rough edges with locking for dead nodes, but it's def…
Does kubernetes not have a mechanism for defining host/rack diversity constraints?
I'm not sure of the current status, but I believe the mechanism would be to label nodes with rack information. You'd then be able to specify anti-affinity constraints based on those labels that would be taken into account by the scheduler.
Edit: browsing issues briefly, I don't think this functionality is available just yet. Discussion of node selectors: https://github.com/kubernetes/kubernetes/issues/341#issuecom... -
Re: I Want to Run Stateful Containers, Too
#20Why not just run lxc and lxd?