Live data from Hacker News

Lessons learned from using Docker Swarm mode in production

blog.bugsnag.com

21–30 of 55 posts

Re: Lessons learned from using Docker Swarm mode in production

#21
post #18
post #16

While we're on the topic can anyone recommend a system for rolling out (Java) applications across server farms that doesn't use containers? We have a bunch of shell scripts that are pretty horrible. We could containerize, but we dont need that right now.

Ansible. It isn't perfect, but is far better than shell scripts for application deployment.

This is what I use. Other players include Puppet, Salt, and Chef

Re: Lessons learned from using Docker Swarm mode in production

#22

Earlier quoted context omitted.

Interesting, thanks! To be honest I haven't looked too much into Kubernetes so far because of the emphasis in cloud deployment, while my interest is in setting it up in bare metal. This feature might make me give it a try though!

There are plenty of people running Kubernetes on bare-metal. There are a number of resources out there for PXE-booting a Kubernetes cluster on bare-metal. Might be time to take a closer look ;).

Yes, but the instructions that I've found to do so are based on installation via shell scripts, which I'm not comfortable with. I'd like to understand it well before running it even in a test setup.

Re: Lessons learned from using Docker Swarm mode in production

#23
post #16

While we're on the topic can anyone recommend a system for rolling out (Java) applications across server farms that doesn't use containers? We have a bunch of shell scripts that are pretty horrible. We could containerize, but we dont need that right now.

I haven't used it, but Nomad has specific support for Java apps.

Re: Lessons learned from using Docker Swarm mode in production

#25

Earlier quoted context omitted.

There are plenty of people running Kubernetes on bare-metal. There are a number of resources out there for PXE-booting a Kubernetes cluster on bare-metal. Might be time to take a closer look ;).

Yes, but the instructions that I've found to do so are based on installation via shell scripts, which I'm not comfortable with. I'd like to understand it well before running it even in a test setup.

The top result for "PXE Boot Kubernetes" is an extremely comprehensive step-by-step guide for it from CoreOS. I didn't see any magic shell scripts. The second top link is Ansible with links the constituent Ansible playbook.

If you want insight into how the pieces fit together, the learnings from Kelsey Hightower's Kubernetes the Hard Way will certainly still map to bare-metal environments as well.

Re: Lessons learned from using Docker Swarm mode in production

#26
> This might be a wishlist item (since we don’t find ourselves doing it frequently enough to merit an automated solution), but it would be very nice to be able to simply bake a new AMI, the completion of which would trigger a job that could swap out instances one or several at a time, such that we would be able to perform zero-downtime upgrades automatically. This can still be done, but right now it’s by hand.

BOSH[0] does rolling deploys, with canaries, out of the box.

At Pivotal we completely upgrade Pivotal Web Services to the latest Cloud Foundry within about a day of it releasing. PWS is our dogfooding the hard way: with a flagship platform that some of our customers have sue-you-to-dust-if-it-fails support contracts for.

Thousands of apps, tens of thousands of containers, thousands of VMS.

None of whom know that we restarted the entire infrastructure beneath them.

Disclosure: I guess that to the degree that Docker Inc realises that platforms are where the money is, my employers at Pivotal are competitors. But BOSH is still a fit for what you want.

[0] http://bosh.io/

Re: Lessons learned from using Docker Swarm mode in production

#27
This is a great write up, thanks for sharing the lessons learned.

I wonder if the open questions about instance management are solved by the "Docker for AWS" beta.

We are entering the commodity phase for orchestration software.

Blogs and HN comments are full of success stories on Swarm, Kubernetes, Mesos, Nomad and ECS.

There are also a few warnings, like the routing issue in this review, but it's simply a matter of time before those get sorted.

What's really going on here is that we are all learning how to handle complexities of distributed systems in the cloud. These new foundations means we can run more sophisticated apps easier and more reliably.

Re: Lessons learned from using Docker Swarm mode in production

#28

> This might be a wishlist item (since we don’t find ourselves doing it frequently enough to merit an automated solution), but it would be very nice to be able to simply bake a new AMI, the completion of which would trigger a job that could swap out instances one or several at a time, such that we would be able to perform zero-downtime upgrades automatically. This can still be done, but right now it’s by hand. BOSH[0…

I respectfully disagree.

When running on AWS you want CloudFormation controlling an AutoScaling Group for this.

Re: Lessons learned from using Docker Swarm mode in production

#29
post #6
post #2

I am a broken record lately on here... Nomad is really awesome for avoiding configuration hell and having to manage multiple services for container orchestration. It's a single binary and very very easy to setup and run. I actually prefer it to Swarm but YMMV. It uses Consul under the hood and has so far been bulletproof. (They all have their drawbacks / idiosyncrasies). https://www.nomadproject.io/

The one feature that to me seems to be essential but appears to be missing from all these container orchestrators is the ability to tie a remote volume (Ceph/Gluster/Lustre/etc) to a container so that if a container is scheduled to run on a certain node, the volume will automatically be mounted on the same node. It seems from the mailing list that at least Nomad will have that at some point, but I have not seen much…

EFS on AWS is an interesting approach. It's a NFS mount that every instance and every container can share.

I wouldn't run a database off it but it's been great for simple file synchronization across containers and container restarts.

Re: Lessons learned from using Docker Swarm mode in production

#30

Honestly, you should try kubernetes. The experience is pretty much the same and the feature set is much more mature.

My experience with Kubernetes is limited, so take all this with a grain of salt. I've been using the integrated swarm since beta.

The new integrated swarm is a real game-changer in that it is much simpler to use compared to other solutions. With swarm, it's simply:

    docker swarm init
    docker swarm join --token    
That being said, I found that Kubernetes offers more granularity in the level of control over the cluster. That's not something that __I__ need necessarily, though obviously YMMV.
Post reply on HN