Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Want to Run Stateful Containers, Too

techcrunch.com

Post reply on HN
Print view View original

61–70 of 102 posts

Re: I Want to Run Stateful Containers, Too

#61

Post by tinco »

stephenitis wrote: post #42
tinco wrote: post #2

We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the functionality we wanted. Like the article author I feel there's some cultural divide happening. How is it possible that the 100.000 line Java project of Kubernetes lacks even basic resource management that our orchestration tool that's written in a few hundred…

I'm curious what was the responsibilities of your own orchestrater? what are examples of situations it had to handle?

Well basically the same things most frameworks do. Encode relations between containers, and allow use to spin containers up on machines, and automatically establishing the links between them. In addition to that, to encode persistent storage resources and link them to tasks. For example machine-1 has 1 HDD of 2TB and 3 SSD's of 250GB. On SSD1 the task postgres-1 has a data resource allocated called abc123. Now when we restart the machine, we can restart postgres-1 task and point it at its data resource and everything will work again.

Note that the orchestrators do sort of give solutions to this issue. They might for example recommend doing the storage inside the docker container, and have the data disappear whenever the container disappears. This might be a little less reliable or transparent, but since you have a redundant cluster you can always restore the data from some other container. Our data is a little bit too big for restoration processes to happen during normal operations. And we're a bit too dependent on the speed to have no control over on which disk a resource is stored.

Re: I Want to Run Stateful Containers, Too

#62

Post by tinco »

sagichmal wrote: post #4
tinco wrote: post #2

We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the functionality we wanted. Like the article author I feel there's some cultural divide happening. How is it possible that the 100.000 line Java project of Kubernetes lacks even basic resource management that our orchestration tool that's written in a few hundred…

Kubernetes is not a Java project.

Sorry got confused, Mesos is C++ and Java project I think. Probably got it swapped with that one.

Re: I Want to Run Stateful Containers, Too

#63

Post by m-app »

vinceguidry wrote: post #17
jacques_chester wrote: post #7

I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own. But this is the bit that surprises me: They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. $1500 per month, versus weeks of engineer…

Yes, I think devs sometimes forget that this is the bleeding-est of bleeding edges. Hell, I can't even get a normal Digital Ocean server instance to programmatically drop and recreate doing minimal things like creating a user and setting SSH keys. I suppose if I wanted to learn something heavyweight like Chef or Puppet I could, but for now, I just maintain a list of commands to run, and understand that Digital Ocean…

> Hell, I can't even get a normal Digital Ocean server instance to programmatically drop and recreate doing minimal things like creating a user and setting SSH keys. I suppose if I wanted to learn something heavyweight like Chef or Puppet I could, but for now, I just maintain a list of commands to run, and understand that Digital Ocean servers can't quite be treated like livestock just yet, though I can fake it pretty well. Eventually Digital Ocean will get around to making the platform changes, which are all already in feature request.

It sounds like you might benefit from Cloud-config for droplets: https://www.digitalocean.com/community/tutorials/an-introduc...

Or have you tried that already? It's also available through their API, BTW.

Re: I Want to Run Stateful Containers, Too

#64

Post by lobster_johnson »

Lennie wrote: post #15
ownagefool wrote: post #8

The author is correct. We should be running stateful containers, but until the the software is ready the default advice quite rightly should be not to unless you understand the implications. For example we run ceph and use it in kubernetes. We obviously only run replica sets (the kubernetes feature, for those uninitiated) of 1 for these services. There's also some rough edges with locking for dead nodes, but it's def…

There are still a whole lots of pieces of the puzzle missing. I think it's all about creating ready to use containers which are running on different hosts which can replicate their data between themselves and handle failovers (they need to connect to etcd or zookeeper or consul for that). For PostgreSQL it seem to have already solved (?) 2 times by different people: https://github.com/zalando/patroni https://flynn.io…

Can anyone explain why Docker image handling is so terrible? For one, building an image creates lots of unnecessary overlays (resulting in the absurd need for "garbage collection"), rather than just compacting everything into one file? But also the whole registry thing. Why can't I just point Docker at an S3 bucket? Sure, you can run a private registry, but that involves (last I checked) running a daemon, Redis, and using "docker login" to get credentials set up... instead of, you know, a file path (like an NFS volume), an HTTP URL, or an S3 bucket. The registry seems like a nice thing for publishing public images, but it has nothing to do with private images.

Re: I Want to Run Stateful Containers, Too

#65

Post by jacques_chester »

bjterry wrote: post #54
jacques_chester wrote: post #29

Earlier quoted context omitted.

> The CloudFoundry installation instructions alone makes me run away screaming. BOSH is getting some much-needed love right now for this reason. Incidentally, for long-running stateful services, BOSH is the right tool for the job. We use it for Cloud Foundry, MySQL and a bunch of others I don't recall right now. Anyhow. If you just want to see quickly if CF fits your needs, the place to go is Lattice[1], explicitly d…

It reads as an extended advertisement for you company's services, and your comment doesn't really respond to the statement of the person you replied to. He said CloudFoundry's installation instructions were bad, then you replied as if he said he was unsure if CF "fit his needs" and was looking for more ways to try it out, which is a pretty salesy response. I think mostly it's a question of tone and length. If you had…

Thanks. I was too emotionally invested.

I actually work for Pivotal Labs, not Pivotal Cloud Foundry (though as of this week I'm on secondment to a PCF-related team).

As a consulting engineer I get to see a lot of projects in a lot of companies. My eagerness for PaaSes comes from seeing a variety of approaches. Just using a PaaS makes large, expensive, disruptive discussions simply disappear.

Those who go through my history will note that I take pains to mention other PaaSes, usually Heroku and OpenShift. When I talk about hosted Cloud Foundry I usually namecheck ours and IBM's.

Re: I Want to Run Stateful Containers, Too

#66

Post by bonobo3000 »

What exactly does containerizing everything give you anyways? I really don't get it.

Before - use chef/puppet to manage dependencies, distribute config files. run processes. maybe use something like upstart to restart on failure.

After - use Dockerfiles to manage dependencies (same thing, bunch of install commands). now you have a container for the web app, one for another service, etc. so everything is isolated.. great. what do you gain over running 2 separate processes? thats pretty damn isolated too.. except for the same disk, they each have own virtual memory, state, config files etc..

I'm not a (modern) ops expert at all, but i know my way around the command line. What do you gain from Docker or say launching a mongo instance on the cloud instead of just renting a server and launching the process? I really want to know. Atleast on a small scale, say if you're managing say 10-20 servers, i don't see the point.

Re: I Want to Run Stateful Containers, Too

#67

Post by Scarblac »

bonobo3000 wrote: post #66

What exactly does containerizing everything give you anyways? I really don't get it. Before - use chef/puppet to manage dependencies, distribute config files. run processes. maybe use something like upstart to restart on failure. After - use Dockerfiles to manage dependencies (same thing, bunch of install commands). now you have a container for the web app, one for another service, etc. so everything is isolated.. gr…

I see an instance as sort of a compiled binary of your app and the environment it runs in. You can install your app, all its dependencies, get all the config files correct, run tests on the created instance, and then you have a Docker instance and can start up any number of them in production.

If any of those steps fails in the middle of some script, you haven't put any server in some half way there state.

If a rollback is needed, you can switch to the previous Docker instance, and changed requirements won't trip you up.

We used to do it with zipped chroot environments and some startup/shutdown scripts, Docker is more or less that.

Of course you need to store data outside them, as otherwise you lose once you switch to a newer version of your instance, but that's easy enough.

All that said, still not really a fan of it.

Re: I Want to Run Stateful Containers, Too

#68

Post by Akkifokkusu »

arbitrarytech wrote: post #41

Not sure where those estimated costs for running MongoDB on AWS came from. It jumps from a single t2.micro instance (which you can get for free) straight to 3x m3.2xlarge instances at $1500/month. That's a pretty big jump. There are at least 6 instance types between the two. Like 3x m3.medium instances with 500GB gp2 EBS volumes would cost $300/month. That's the on-demand pricing, you could probably save some more wi…

The chart comes from this AWS-provided document, actually: https://s3.amazonaws.com/quickstart-reference/mongodb/latest...

Re: I Want to Run Stateful Containers, Too

#69

Post by dpeterson »

bonobo3000 wrote: post #66

What exactly does containerizing everything give you anyways? I really don't get it. Before - use chef/puppet to manage dependencies, distribute config files. run processes. maybe use something like upstart to restart on failure. After - use Dockerfiles to manage dependencies (same thing, bunch of install commands). now you have a container for the web app, one for another service, etc. so everything is isolated.. gr…

Even on a small scale there are significant benefits, both in production and in development and test. As the article states, containers are great for moving away from a monolith architecture. Many applications are moving to a distributed service oriented architecture. Each piece can be deployed separate. That leads to an easier more maintainable development process. However, the larger number of independently deployable apps adds operation complexity. Trying to manage that complexity, even on a small scale is no fun. Queue containers. Containers by themselves are interesting; they let you run many more applications on a single machine. Unlike a VM, there is no operating system chewing up ram and cpu. On machines only able to run 4 VM's reasonably well, I have seen nearly a hundred containers. That is only a small part. Container platforms such as Openshift make the process much nicer and more powerful. Openshift uses Kubernetes under the covers. You arrange your applications inside pods. The pods are wrapped in a Kubernetes service. The pods automatically assign internal ip addresses; that allows you to put, let's say, 20 wildfly servers on the same machine without worrying about ip and port clashes. The service acts as a load balancer and a router directs traffic accordingly. I don't have to worry about any of that. It is all automated. I create a build config and a deployment config for my services (there could be hundreds) then Openshift (or other container PAAS) maximizes the potential of my hardware. In development, it is much easier to completely replicate production. Everything is encapsulated in the container image. Not just the application; all of the software and configuration the application requires is part of the image. If I need to recreate the container image, every step is in the Dockerfile. I do not need documentation to remind me how everything is set up. In openshift, I grab a template, then click a button; within seconds, the containers of my app are running. That would take much longer with VMs. Containers allow me to develop in a production like environment because I am able to deploy an unscaled version easily. What took multiple physical machines to replicate now takes a few seconds to deploy on a laptop.

Re: I Want to Run Stateful Containers, Too

#70

Post by nullspace »

stephenitis wrote: post #50
siliconc0w wrote: post #40

If you're using AWS you can use a pre-task (i.e fleet unit file) to run something like https://github.com/leg100/docker-ebs-attach to attach an volume before running your container. You can also do this with Flocker( https://docs.clusterhq.com/en/1.7.2/config/aws-configuration... ) if you want something fancier. Still rather have AWS manage the data. Unless you're a really biggie sized company RDS/Elasticache are rea…

I'm curious Would you be ok with being locked in with X provider's storage specific solution? (say a ECS only way of doing things) Is the headache in the setup, config, or risk of being at the helm of orchestrating your own data?

My experience is that you are going to be "locked in" in some way no matter what. Current infrastructure systems are a mess of vendor specific solutions and configurations. Migrating from one open source system to another is going to be just as hard as migrating from a proprietary thing like vanilla ECS to say kubernetes on bare metal.

There are other concerns like vendor pricing and stuff, but I have not had bad luck with that.

Post reply on HN