2k nodes, 100k containers https://blog.online.net/2016/07/29/docker-swarm-an-analysis-...
5k nodes, 1 million containers, 5 minutes https://www.hashicorp.com/c1m/
(Disclaimer: I'm on the Nomad team but wasn't at the time of the post)
Running 1000 containers in Docker Swarm
11–20 of 37 posts
Re: Running 1000 containers in Docker Swarm
#12I would use IPv6 for the orchestration network, probably not touch the tcp/ip parameters except for port range (and open file descriptor), and break up the broadcast domain into smaller networks. It is not advisable to have thousands of machines on one broadcast domain, and it is a pain in the ass to troubleshoot, not to mention causes bigger headaches when one network problem affects all the nodes across the entire gigantic network.
Re: Running 1000 containers in Docker Swarm
#132k nodes, 100k containers https://blog.online.net/2016/07/29/docker-swarm-an-analysis-...
Since we're playing this game:
1.55k nodes, 250k containersed applications[0].
Mind you, it's hard to compare these as there's no real "cloud bench". For pure benchmark porn Nomad are the undisputed champs on their 1 million case.
The Cloud Foundry scaling test was intended to show a system with fully service-configured, fully-routed apps, with varying app characteristics (memory and RPS). To further stress the system, thousands of apps crashing and are relaunched on a continuous basis.
Cloud Foundry installations with >10k containers have been ordinary for a while now; the 250k thing was to ensure we had lots of headroom and shake out chokepoints in Diego.
[0] https://content.pivotal.io/blog/250k-containers-in-productio...
Disclosure: I work for Pivotal, the majority donor of engineering on Cloud Foundry.
Re: Running 1000 containers in Docker Swarm
#14Slight nitpick, but this articles deals with "Docker Swarm mode" [1], which is different from Docker Swarm [2].
[1] https://docs.docker.com/engine/swarm/
[2] https://github.com/docker/swarm
[3] Difference between Docker Swarm and Swarm mode: http://stackoverflow.com/questions/40039031/what-is-the-diff...
Re: Running 1000 containers in Docker Swarm
#15Does anyone know how easy it is to set up autoscaling with Docker Swarm running on Google Cloud or AWS? We're looking to get starting with Docker Swarm or Kubernetes soon, and are considering using Docker Swarm because of its simplicity and developer familiarity with Docker Compose (we use it for our dev environment). We just want to add nodes to a cluster as traffic spikes and subsides.
In the context of Docker Swarm and Kubernetes, autoscaling refers to container level scaling ie. given a set of nodes, any autoscaling function would manage the number of containers that are currently running on these nodes.
For instance/node level autoscaling (which is closer to what you need), I would recommend using the autoscaling features provided by AWS/Google Cloud.
Re: Running 1000 containers in Docker Swarm
#16I always wonder, why not isolate on a process level, or even withing a single, multi-threaded app. Sure you can run some sort of web service on hundreds of docker containers or you can run a single, fast web server that scales?
When that single web server goes down, it's not so "fast" anymore.
That sounds like a fixable problem. I'm pretty sure Erlang programmers could give some tips.
Why is worrying about a single web server going down more worrisome than some part of the Docker stack going down and causing the same issue?
Re: Running 1000 containers in Docker Swarm
#17I always wonder, why not isolate on a process level, or even withing a single, multi-threaded app. Sure you can run some sort of web service on hundreds of docker containers or you can run a single, fast web server that scales?
When that single web server goes down, it's not so "fast" anymore.
Agreed, though I keep wanting to take the time to get VRRP working with a web server to have redundancy. OpenBSD uses this to coordinate stateful firewalls with 2 or more systems, if 1 goes down all state info is present on the second node which takes over.
Re: Running 1000 containers in Docker Swarm
#18Does anyone know how easy it is to set up autoscaling with Docker Swarm running on Google Cloud or AWS? We're looking to get starting with Docker Swarm or Kubernetes soon, and are considering using Docker Swarm because of its simplicity and developer familiarity with Docker Compose (we use it for our dev environment). We just want to add nodes to a cluster as traffic spikes and subsides.
I think most people would suggest that, if your use case is at a stage where that is important to you, Swarm is not the right thing. Kubernetes or ECS are better choices.
Re: Running 1000 containers in Docker Swarm
#19I always wonder, why not isolate on a process level, or even withing a single, multi-threaded app. Sure you can run some sort of web service on hundreds of docker containers or you can run a single, fast web server that scales?
Because that wouldn't allow you to own a unicorn-size startup providing tools and technology for containerization?
Re: Running 1000 containers in Docker Swarm
#20Earlier quoted context omitted.
When that single web server goes down, it's not so "fast" anymore.
Agreed, though I keep wanting to take the time to get VRRP working with a web server to have redundancy. OpenBSD uses this to coordinate stateful firewalls with 2 or more systems, if 1 goes down all state info is present on the second node which takes over.
Hi, OP author here: I have actually set up a VRRP (well, UCARP) on Docker, so it's possible even to containerize this facet of running a HA ops stack with Docker as the infrastructure. It is however, as you say, it is only used for one active node + a number of fail-overs in case that one goes down. In terms of maintenance (hosts do go down, scheduled downtime is common), it's priceless to have this part of the puzzle portable as well. If you want to check it out, there's a github available here: https://github.com/titpetric/ucarp-ha - and a future article with it is planned as well. It will also become a part of the E-book which I'm currently working on and publishing on leanpub: https://leanpub.com/12fa-docker-golang :)