Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running 1000 containers in Docker Swarm

blog.codeship.com

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Running 1000 containers in Docker Swarm

#21

Post by patrickg_zill »

titpetric wrote: post #20
patrickg_zill wrote: post #17

Earlier quoted context omitted.

Agreed, though I keep wanting to take the time to get VRRP working with a web server to have redundancy. OpenBSD uses this to coordinate stateful firewalls with 2 or more systems, if 1 goes down all state info is present on the second node which takes over.

Hi, OP author here: I have actually set up a VRRP (well, UCARP) on Docker, so it's possible even to containerize this facet of running a HA ops stack with Docker as the infrastructure. It is however, as you say, it is only used for one active node + a number of fail-overs in case that one goes down. In terms of maintenance (hosts do go down, scheduled downtime is common), it's priceless to have this part of the puzzl…

OK so that runs on the host to the Docker instances. Pretty cool!

Re: Running 1000 containers in Docker Swarm

#22

Post by titpetric »

undersuit wrote: post #16
acejam wrote: post #8

Earlier quoted context omitted.

When that single web server goes down, it's not so "fast" anymore.

That sounds like a fixable problem. I'm pretty sure Erlang programmers could give some tips. Why is worrying about a single web server going down more worrisome than some part of the Docker stack going down and causing the same issue?

Actually, neither should be a problem if you have enough redundancy :) the hardest part of rolling your own infrastructure is testing mission critical systems (like databases) to be fault tolerant and at the same time reliable. Lots of great projects are out there that address some of these issues, but it takes a lot of attention to details (like transaction rates, ACID compliance, replication, etc.) to get it right. This is why a lot of developers which aren't in unicorn startups take advantage of technology which is available from giants like Amazon or Google, or specific problem-domain companies like CloudFlare for example. Netflix serves as a great example of a technology-driven company that is an inspiration to us, but there are so many others that really changed the way we approach problems - Tumblr, Etsy. But to stay on topic of netflix - I think their idea behind "chaos monkey" is great, and we're increasingly rolling out a (currently simple) docker swarm version of it - https://github.com/titpetric/docker-chaos-monkey - the best way to eliminate worry is to test failure scenarios. As docker chaos monkey is designed to unpredictably "kill off" containers, your system gets the benefit of design to handle failures. It's one of those problems that you have to have a passion for however - it's like testing software. You're only testing software for the functionality and failures which you can predict, and I'm pretty sure that any of us can't predict all the ways in which software (or distributed systems) can fail. As such, it's a never ending occupation. :)

Re: Running 1000 containers in Docker Swarm

#23

Post by jacques_chester »

schmichael wrote: post #11
barhun wrote: post #2

2k nodes, 100k containers https://blog.online.net/2016/07/29/docker-swarm-an-analysis-...

5k nodes, 1 million containers, 5 minutes https://www.hashicorp.com/c1m/ (Disclaimer: I'm on the Nomad team but wasn't at the time of the post)

I don't know much about Nomad and couldn't work out from the repo what the jobs were. If I guess correctly, it's an app using Redis. Is that correct?

Disclosure: by coincidence of market forces, we're mortal enemies. Let's send christmas cards!

Re: Running 1000 containers in Docker Swarm

#24

Post by jacques_chester »

xchaotic wrote: post #4

I always wonder, why not isolate on a process level, or even withing a single, multi-threaded app. Sure you can run some sort of web service on hundreds of docker containers or you can run a single, fast web server that scales?

Sometimes you have endpoint A and endpoint B.

They are part of the same app.

They should not have the same level of privilege.

The secrets in endpoint A's memory should not be visible to endpoint B and vice versa.

Containers increase assurance that this is so.

Re: Running 1000 containers in Docker Swarm

#25

Post by merb »

jacques_chester wrote: post #24
xchaotic wrote: post #4

I always wonder, why not isolate on a process level, or even withing a single, multi-threaded app. Sure you can run some sort of web service on hundreds of docker containers or you can run a single, fast web server that scales?

Sometimes you have endpoint A and endpoint B. They are part of the same app. They should not have the same level of privilege. The secrets in endpoint A's memory should not be visible to endpoint B and vice versa. Containers increase assurance that this is so.

systemd can do the same thing. the only thing that docker adds or containers are immutability.

Re: Running 1000 containers in Docker Swarm

#26

Post by titpetric »

jacques_chester wrote: post #13
barhun wrote: post #2

2k nodes, 100k containers https://blog.online.net/2016/07/29/docker-swarm-an-analysis-...

Since we're playing this game: 1.55k nodes, 250k containersed applications[0]. Mind you, it's hard to compare these as there's no real "cloud bench". For pure benchmark porn Nomad are the undisputed champs on their 1 million case. The Cloud Foundry scaling test was intended to show a system with fully service-configured, fully-routed apps, with varying app characteristics (memory and RPS). To further stress the syste…

Big respect for your achievements. I guess at some point it just becomes the question of "where do i get a 1000 nodes" vs. "how do I run a 1000 containers". Or, more the justification for that amount of hardware - I mean, the one dream job which I would probably want is getting paid to cut out all the hardware use while keeping reliability/availability/functionality. Like these guys who cut their AWS bill by $1mil/year in about 3 months - https://segment.com/blog/the-million-dollar-eng-problem/. The thing is that I'm not exactly sure where I'd fit in more - running this thing, or just fixing it for somebody else. I definitely know that I'm mostly dealing with pets and not cattle :)

Re: Running 1000 containers in Docker Swarm

#27

Post by schmichael »

jacques_chester wrote: post #23
schmichael wrote: post #11

Earlier quoted context omitted.

5k nodes, 1 million containers, 5 minutes https://www.hashicorp.com/c1m/ (Disclaimer: I'm on the Nomad team but wasn't at the time of the post)

I don't know much about Nomad and couldn't work out from the repo what the jobs were. If I guess correctly, it's an app using Redis. Is that correct? Disclosure: by coincidence of market forces, we're mortal enemies. Let's send christmas cards!

> it's an app using Redis. Is that correct?

Yup! Repo could definitely be clearer, but here's the code:

https://github.com/hashicorp/c1m/blob/master/schedbench/test...

Basically calls an increment in Redis and then blocks forever.

> Disclosure: by coincidence of market forces, we're mortal enemies. Let's send christmas cards!

Haha, hi mortal enemy! Christmas cards it is! If you're ever in Portland, OR I'll buy a beverage of your choice as well. :)

Re: Running 1000 containers in Docker Swarm

#28

Post by eblanshey »

hefeweizen wrote: post #15
eblanshey wrote: post #3

Does anyone know how easy it is to set up autoscaling with Docker Swarm running on Google Cloud or AWS? We're looking to get starting with Docker Swarm or Kubernetes soon, and are considering using Docker Swarm because of its simplicity and developer familiarity with Docker Compose (we use it for our dev environment). We just want to add nodes to a cluster as traffic spikes and subsides.

In the context of Docker Swarm and Kubernetes, autoscaling refers to container level scaling ie. given a set of nodes, any autoscaling function would manage the number of containers that are currently running on these nodes. For instance/node level autoscaling (which is closer to what you need), I would recommend using the autoscaling features provided by AWS/Google Cloud.

> I would recommend using the autoscaling features provided by AWS/Google Cloud

It would have to be integrated with Kubernetes though -- when we push a new docker container, the container would need to be updated on any new machines created. We'll look into GCP's autoscale solution.

Re: Running 1000 containers in Docker Swarm

#29

Post by dalailambda »

eblanshey wrote: post #28
hefeweizen wrote: post #15

Earlier quoted context omitted.

In the context of Docker Swarm and Kubernetes, autoscaling refers to container level scaling ie. given a set of nodes, any autoscaling function would manage the number of containers that are currently running on these nodes. For instance/node level autoscaling (which is closer to what you need), I would recommend using the autoscaling features provided by AWS/Google Cloud.

> I would recommend using the autoscaling features provided by AWS/Google Cloud It would have to be integrated with Kubernetes though -- when we push a new docker container, the container would need to be updated on any new machines created. We'll look into GCP's autoscale solution.

The node level autoscaling doesn't need to be integrated with kubernetes, all it needs to do is create a new instance and register it as a node through normal channels.

Even if you don't need autoscaling, I'd suggest still using autoscaling groups and setting it to a fixed number of instances, so that instances will automatically get restarted if they go down.

Re: Running 1000 containers in Docker Swarm

#30

Post by hefeweizen »

eblanshey wrote: post #28
hefeweizen wrote: post #15

Earlier quoted context omitted.

In the context of Docker Swarm and Kubernetes, autoscaling refers to container level scaling ie. given a set of nodes, any autoscaling function would manage the number of containers that are currently running on these nodes. For instance/node level autoscaling (which is closer to what you need), I would recommend using the autoscaling features provided by AWS/Google Cloud.

> I would recommend using the autoscaling features provided by AWS/Google Cloud It would have to be integrated with Kubernetes though -- when we push a new docker container, the container would need to be updated on any new machines created. We'll look into GCP's autoscale solution.

Yeah, any new machine instance has to join the Swarm (and its equivalent in kubernetes-speak). But that can be decoupled from kubernetes or docker swarm mode.

As for image management, it would depend on how you would like to propagate new images. With a private docker registry, you could potentially point each new instance to the registry and take care of propagating new images. I favor this approach since it keeps everything separate and easier to manage.

Post reply on HN