I Want to Run Stateful Containers, Too
techcrunch.com
I Want to Run Stateful Containers, Too
1–10 of 102 posts
Re: I Want to Run Stateful Containers, Too
#2We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the functionality we wanted. Like the article author I feel there's some cultural divide happening. How is it possible that the 100.000 line Java project of Kubernetes lacks even basic resource management that our orchestration tool that's written in a few hundred lines of bash and Ruby does have?
Re: I Want to Run Stateful Containers, Too
#3I think the author is implicitly describing the paradigm shift from software and hardware abstractions to service abstractions. We used to make computers by soldering electronics together, then by assembling cases and components, then buying premade servers, then renting cloud capacity, and now we're starting to rent everything as services.
The cloud is about going from capex to opex and the "capex" now is the initial work needed to define your own architectures and stacks for every project (before you get to work on the actual project, i.e. the differentiating part). Amazon is eliminating most of this by offering building block services that fit together with little hassle.
So the challenge for open source is how to move on to this era of service abstraction. It's no longer enough to just provide an NPM package or a configure script.
I think Docker is in a good position to bring us there, but it's currently stuck at the stateless container level. Something needs to evolve so that launching a scalable and auto-maintainable database cluster along with a connected web application cluster is as easy with Docker as it is by renting a few Amazon service.
Re: I Want to Run Stateful Containers, Too
#4We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the functionality we wanted. Like the article author I feel there's some cultural divide happening. How is it possible that the 100.000 line Java project of Kubernetes lacks even basic resource management that our orchestration tool that's written in a few hundred…
Kubernetes is not a Java project.
Re: I Want to Run Stateful Containers, Too
#5I think the author is implicitly describing the paradigm shift from software and hardware abstractions to service abstractions. We used to make computers by soldering electronics together, then by assembling cases and components, then buying premade servers, then renting cloud capacity, and now we're starting to rent everything as services. The cloud is about going from capex to opex and the "capex" now is the initia…
How does OpenStack stack up in this regard? A step in the right direction?
Re: I Want to Run Stateful Containers, Too
#6We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the functionality we wanted. Like the article author I feel there's some cultural divide happening. How is it possible that the 100.000 line Java project of Kubernetes lacks even basic resource management that our orchestration tool that's written in a few hundred…
* Kubernetes is not written in Java
* There is no one size fit all, just like never ending Ruby vs JavaScript VS Java.
* Kubernetes just hit v1.0 not long ago.
* Seem like you already wrote the functionality you want, now what is the problem?
Re: I Want to Run Stateful Containers, Too
#7I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own.
But this is the bit that surprises me:
They get you hooked for free and the next level
is $1,496 per month… wtf! MongoLabs is little
better.
Supposing the author was running on Pivotal Web Services. It would've taken less time to add a MongoLab service (about 2 minutes, 3 if you include a re-stage) than to perform the CloudFormation calculation (say, half an hour, resulting in no running software).
PaaSes like Heroku, Cloud Foundry and OpenShift are feature-complete for the cases that application engineers and operators care about. If you roll your own you're directing effort to something that doesn't provide user value.
If I walked in on someone at a regular dev company rolling their own operating system for a web service, I'd be surprised. Writing their own programming language? I'd be skeptical. Oh you built a new HTTP server? Why? Outside of a research environment, why are you doing that?
And so it is with PaaSes. The marker was passed years ago. We don't need to go on these spiritual quests any more.
Disclaimer: I work for Pivotal, which donates the majority of engineering effort on Cloud Foundry. I'm actually in Pivotal Labs, the agile consulting division, which is where I morphed into a lay preacher for just-using-a-PaaS-dammit.
Re: I Want to Run Stateful Containers, Too
#8The author is correct. We should be running stateful containers, but until the the software is ready the default advice quite rightly should be not to unless you understand the implications.
For example we run ceph and use it in kubernetes. We obviously only run replica sets (the kubernetes feature, for those uninitiated) of 1 for these services. There's also some rough edges with locking for dead nodes, but it's definitely good enough for running the likes of Jira or Gitlab have single points of failure pretty much whatever way you scale them (unless you build a HA nfs server first).
Before that, we were running test clusters with just fleet, using custom bash and etcd for service discovery. Works fine, but you're probably better off with frameworks.
Now where things can go horribly wrong is you decide to use a clustered database, run it in kubernetes, thinking it's fine since you can't mount external storage for replica's but you have 3 nodes over 3 zones and you have backups. Kubernetes will quite happily schedule all of your replica's on the same host, meaning a single reboot and you're dead. There are hacks around this such as defining a hostPort, but of course if someone updates the RC you could happily be restoring to backup. You could run two multiple replica controllers to get around this and that'll allow you to mount storage again but that takes a way a lot of the elegance of the framework.
Point is, you can do it, but you'll need a bunch of uglyness and you'll need to be careful. If that doesn't sound great, way until the software properly supports it. :)
Re: I Want to Run Stateful Containers, Too
#9I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own. But this is the bit that surprises me: They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. $1500 per month, versus weeks of engineer…
> $1500 per month, versus weeks of engineering time spent tinkering with and upgrading and bug-fixing and trouble-shooting and security-patching a hand-made solution is a fantastic bargain.
That's only true if you look at it from the point of view of an enterprise, a startup with VC funding, or generally a "rich" company. If you're a cash-strapped startup/small business with only a few thousand dollars profit per month, $1500/mo is a huge deal.
Re: I Want to Run Stateful Containers, Too
#10I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own. But this is the bit that surprises me: They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. $1500 per month, versus weeks of engineer…
> $1500 per month, versus weeks of engineering time spent tinkering with and upgrading and bug-fixing and trouble-shooting and security-patching a hand-made solution is a fantastic bargain. That's only true if you look at it from the point of view of an enterprise, a startup with VC funding, or generally a "rich" company. If you're a cash-strapped startup/small business with only a few thousand dollars profit per mon…
Wouldn't need to worry about auto scaling Mongo yourself if you at that stage of the company though? You'd need to have quite a bit of traffic to need it. And if you do, hopefully you have generated $ needed to support it.