Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running 1000 containers in Docker Swarm

blog.codeship.com

Post reply on HN
Print view View original

31–37 of 37 posts

Re: Running 1000 containers in Docker Swarm

#31

Post by jacques_chester »

schmichael wrote: post #27
jacques_chester wrote: post #23

Earlier quoted context omitted.

I don't know much about Nomad and couldn't work out from the repo what the jobs were. If I guess correctly, it's an app using Redis. Is that correct? Disclosure: by coincidence of market forces, we're mortal enemies. Let's send christmas cards!

> it's an app using Redis. Is that correct? Yup! Repo could definitely be clearer, but here's the code: https://github.com/hashicorp/c1m/blob/master/schedbench/test... Basically calls an increment in Redis and then blocks forever. > Disclosure: by coincidence of market forces, we're mortal enemies. Let's send christmas cards! Haha, hi mortal enemy! Christmas cards it is! If you're ever in Portland, OR I'll buy a beve…

I extend the same offer in New York!

Re: Running 1000 containers in Docker Swarm

#32

Post by jacques_chester »

titpetric wrote: post #26
jacques_chester wrote: post #13

Earlier quoted context omitted.

Since we're playing this game: 1.55k nodes, 250k containersed applications[0]. Mind you, it's hard to compare these as there's no real "cloud bench". For pure benchmark porn Nomad are the undisputed champs on their 1 million case. The Cloud Foundry scaling test was intended to show a system with fully service-configured, fully-routed apps, with varying app characteristics (memory and RPS). To further stress the syste…

Big respect for your achievements. I guess at some point it just becomes the question of "where do i get a 1000 nodes" vs. "how do I run a 1000 containers". Or, more the justification for that amount of hardware - I mean, the one dream job which I would probably want is getting paid to cut out all the hardware use while keeping reliability/availability/functionality. Like these guys who cut their AWS bill by $1mil/ye…

Well, Cloud Foundry is deployed by BOSH. So you can, if you wish, use RackHD to deploy it to naked hardware (instead of OpenStack, GCP, AWS, Azure and I forget what else).

Your apps will still be containerised, distributed and wired up the same way.

There's always a point at which it makes engineering sense to flip the switch to doing it yourself. But that frontier is never static. We (plus our peers in the Cloud Foundry Foundation) and others in this space like Red Hat OpenShift are constantly pushing back the tipping point at which it makes economic sense to DIY.

We already have very large customers with very large engineering teams, who've built platforms before. And they are switching because that effort no longer makes business sense. It's an expense they don't need for a platform they're the only maintainers of.

One of our peers at IBM wrote about DIY[0]. We have our own much more markety-businessy whitepaper, with a very detailed case, on the same topic[1].

[0] https://hackernoon.com/stop-spending-engineering-effort-solv...

[1] https://content.pivotal.io/white-papers/the-upside-down-econ...

Disclosure: I work for Pivotal, etc.

Re: Running 1000 containers in Docker Swarm

#33

Post by jacques_chester »

merb wrote: post #25
jacques_chester wrote: post #24

Earlier quoted context omitted.

Sometimes you have endpoint A and endpoint B. They are part of the same app. They should not have the same level of privilege. The secrets in endpoint A's memory should not be visible to endpoint B and vice versa. Containers increase assurance that this is so.

systemd can do the same thing. the only thing that docker adds or containers are immutability.

Cloud Foundry uses Garden which uses runC. But our Garden had a container system that predated docker and nspawn. So probably another case of Not Invented Yet Syndrome.

Re: Running 1000 containers in Docker Swarm

#34

Post by truetuna »

013a wrote: post #18
eblanshey wrote: post #3

Does anyone know how easy it is to set up autoscaling with Docker Swarm running on Google Cloud or AWS? We're looking to get starting with Docker Swarm or Kubernetes soon, and are considering using Docker Swarm because of its simplicity and developer familiarity with Docker Compose (we use it for our dev environment). We just want to add nodes to a cluster as traffic spikes and subsides.

I think most people would suggest that, if your use case is at a stage where that is important to you, Swarm is not the right thing. Kubernetes or ECS are better choices.

I wouldn't recommend ECS. I've used it for a little over 6 months and even for trivial things, it lacks. A couple examples that come to mind include not being able to pass host environment variables into your container instances easily, and not being able to specify that a service must run on all hosts.

Theres an open issue (made ~2 years ago) on GH for the 1st example and it still hasn't been resolved.

Re: Running 1000 containers in Docker Swarm

#36

Post by collyw »

jacques_chester wrote: post #24
xchaotic wrote: post #4

I always wonder, why not isolate on a process level, or even withing a single, multi-threaded app. Sure you can run some sort of web service on hundreds of docker containers or you can run a single, fast web server that scales?

Sometimes you have endpoint A and endpoint B. They are part of the same app. They should not have the same level of privilege. The secrets in endpoint A's memory should not be visible to endpoint B and vice versa. Containers increase assurance that this is so.

Databases have had table level privileges for decades. Not quite the same but its easy enough to use it for the same purpose.

Re: Running 1000 containers in Docker Swarm

#37

Post by jacques_chester »

collyw wrote: post #36
jacques_chester wrote: post #24

Earlier quoted context omitted.

Sometimes you have endpoint A and endpoint B. They are part of the same app. They should not have the same level of privilege. The secrets in endpoint A's memory should not be visible to endpoint B and vice versa. Containers increase assurance that this is so.

Databases have had table level privileges for decades. Not quite the same but its easy enough to use it for the same purpose.

I agree with you.

But if a single process has the single account on the database, how do you partition those permissions? Simply providing multiple logins won't help if you assume hostile code is in your process space.

On the other hand, if each service has its own login, then the database can enforce lowest authority for each. A compromise of one service isn't a game over scenario.

It's the difference between having a single account with the union of all permissions, or disjoint sets.

Post reply on HN