Not sure where those estimated costs for running MongoDB on AWS came from. It jumps from a single t2.micro instance (which you can get for free) straight to 3x m3.2xlarge instances at $1500/month. That's a pretty big jump. There are at least 6 instance types between the two. Like 3x m3.medium instances with 500GB gp2 EBS volumes would cost $300/month. That's the on-demand pricing, you could probably save some more with reservations given the stateful nature of MongoDB.
I Want to Run Stateful Containers, Too
41–50 of 102 posts
Re: I Want to Run Stateful Containers, Too
#42We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the functionality we wanted. Like the article author I feel there's some cultural divide happening. How is it possible that the 100.000 line Java project of Kubernetes lacks even basic resource management that our orchestration tool that's written in a few hundred…
I'm curious what was the responsibilities of your own orchestrater?
what are examples of situations it had to handle?
Re: I Want to Run Stateful Containers, Too
#43So I fought with this over the last two years, and went from running Postgres in a Docker container, to Amazon RDS after getting frustrated with maintaining Docker volumes, and then now, back to using Docker volumes via Kubernetes' volume attachment.
I think Kubernetes has done a great job at tackling this, at least as a first pass. Right now, I can attach an EBS volume as my Postgres data store, and not worry about where the container is running, since Kubernetes handles mounting the volume. Presumably, I can run an NFS server and have it use that instead of an EBS volume.
Now, I can run backups and slaves however I like. It's not as easy as RDS, but I have more control now, and it's marginally cheaper.
Anyway, I see where the author is coming from, and we're not totally there yet, but the problem is being solved.
Re: I Want to Run Stateful Containers, Too
#44I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own. But this is the bit that surprises me: They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. $1500 per month, versus weeks of engineer…
On what planet is it worth paying $20,000 per month for a vm??
Re: I Want to Run Stateful Containers, Too
#45Earlier quoted context omitted.
> $1500 per month, versus weeks of engineering time spent tinkering with and upgrading and bug-fixing and trouble-shooting and security-patching a hand-made solution is a fantastic bargain. That's only true if you look at it from the point of view of an enterprise, a startup with VC funding, or generally a "rich" company. If you're a cash-strapped startup/small business with only a few thousand dollars profit per mon…
This is especially true when you consider what that same $1500/mo will buy you using something like DynamoDB or Aurora. Both those solutions will give you more storage, are managed for you and will mostly scale up with you, meaning you don't have to start anywhere near $1500/mo. I know the article said tying yourself to Amazon feels wrong. But focusing your time and energy on managing infrastructure that could be man…
A big +1 for DynamoDB + S3 for a cash strapped startup.
If you are writing an app from scratch, and the cost of data services is a big concern, then don't pick a data store with complex and expensive replication properties.
DynamoDB is admittedly harder to understand and use than Postgres or Mongo, but when you figure it out its a HTTP data API with no setup or maintenance costs.
$8/mo of DynamoDB can easily cover your users and other CRUD.
Re: I Want to Run Stateful Containers, Too
#46I think the author is implicitly describing the paradigm shift from software and hardware abstractions to service abstractions. We used to make computers by soldering electronics together, then by assembling cases and components, then buying premade servers, then renting cloud capacity, and now we're starting to rent everything as services. The cloud is about going from capex to opex and the "capex" now is the initia…
>The cloud is about going from capex to opex and the "capex" now is the initial work needed to define your own architectures and stacks for every project (before you get to work on the actual project, i.e. the differentiating part). Amazon is eliminating most of this by offering building block services that fit together with little hassle. Maybe they are eliminating it for themselves, but if they are eliminating most…
>there will be little to differentiate your service from any other Amazon hosted/run service.
Besides, you know, your product.
Infrastructure can kill your product if it's awful, but it certainly can't "differentiate" it in a good way.
Re: I Want to Run Stateful Containers, Too
#47The author is correct. We should be running stateful containers, but until the the software is ready the default advice quite rightly should be not to unless you understand the implications. For example we run ceph and use it in kubernetes. We obviously only run replica sets (the kubernetes feature, for those uninitiated) of 1 for these services. There's also some rough edges with locking for dead nodes, but it's def…
There are still a whole lots of pieces of the puzzle missing. I think it's all about creating ready to use containers which are running on different hosts which can replicate their data between themselves and handle failovers (they need to connect to etcd or zookeeper or consul for that). For PostgreSQL it seem to have already solved (?) 2 times by different people: https://github.com/zalando/patroni https://flynn.io…
Flynn co-founder here. We plan to expand to support many more popular open source databases next year (and expose a framework for others to do the same). The goal is that a supported database should "just work" on the platform and be sanely configured out of the box with high availability, automatic failover/recovery, encrypted streaming backups, app credential provisioning, etc.
Our Postgres appliance is unique in that it is explicitly designed to not lose data in the face of failure. Simply wrapping up a database in a container with leader election and replication is not enough, as there are many pitfalls that can cause data to be lost during failures. Of course we are limited by the guarantees the database can provide, so some datastore appliances we build may have clearly defined caveats.
Storing binary blobs is another tricky matter, and we're exploring what we can do with little to no configuration. We currently store app images and git repos in Postgres, which works but will only scale so far.
We are also improving our scheduler so that it is capable of providing the constraints necessary to place stateful services properly.
Re: I Want to Run Stateful Containers, Too
#48The author seems to be mixing some valid observations regarding the difficulty of stateful containers with some paranoia about the growth of cloud deployments and ownership of data. Or maybe it's not paranoia. I don't know. But it's different.
First off, containers don't absolutely have to be stateless. The first and foremost benefit of containers is dependency isolation and configuration management. Once you use them for any length of time this becomes clear. You make them, put them on a machine with a compatible kernel and network access to the right stuff and they just run.
It's a pretty short leap from containers that just run to the idea of container orchestration systems like kubernetes. We just deployed a new staging environment built on Google Container Engine, an implementation of kubernetes, and it's pretty damn amazing what you can do at the services layer, and yes, even at the gateway and persistence layers. But you have to treat the needs of these layers differently.
Statelessness is important at the services layer because ultimately you want to scale up and down seamlessly and automatically, and kubernetes allows you to do just that.
In the persistence layer it's the opposite: state is all that's important and scaling is a more complicated affair. That doesn't mean containers aren't useful in that layer. They still provide the above-mentioned benefits. The aforementioned staging environment uses elasticsearch running in a dedicated kubernetes cluster, where each pod is bolted to a persistent disk at cluster creation. It also uses a mongo replicaset that is just deployed on instances in the old manner, but we have a prototype containerized install and will be moving toward that. Lastly it uses mysql via Google's cloudsql managed offering.
So you have a lot of differences in the persistence layer, and a lot of choices for how to manage those differences. Things are a lot simpler and cleaner at the services layer, but that doesn't mean the benefits of containers in one layer are somehow less of a win than in the other. After three years of using and deploying them my feeling is it's pretty much all win.
Re: I Want to Run Stateful Containers, Too
#49One of huge challenge in solving stateful containeres is being as agnostic to the orchestration tool Docker Swarm, Kubernetes, Mesosphere, etc, all of which have different opinions of how clusters of containers should be orchestrated while also accounting for variety in what hosting a cluster of stateful container(s) means to the user.
I work at ClusterHQ, our team believes the tools we are building like Flocker are going to get the community there.
it's pluggable to both orchestration tools and has a model for creating backend plugins.
Storage backend provider plugins that work with Flocker. http://doc-dev.clusterhq.com/config/configuring-nodes-storag...
Re: I Want to Run Stateful Containers, Too
#50If you're using AWS you can use a pre-task (i.e fleet unit file) to run something like https://github.com/leg100/docker-ebs-attach to attach an volume before running your container. You can also do this with Flocker( https://docs.clusterhq.com/en/1.7.2/config/aws-configuration... ) if you want something fancier. Still rather have AWS manage the data. Unless you're a really biggie sized company RDS/Elasticache are rea…
I'm curious
Would you be ok with being locked in with X provider's storage specific solution? (say a ECS only way of doing things)
Is the headache in the setup, config, or risk of being at the helm of orchestrating your own data?