Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From 30 to 230 Docker containers per host

sven.stormbind.net

Post reply on HN
Print view View original

31–40 of 55 posts

Re: From 30 to 230 Docker containers per host

#31

Post by programd »

sandGorgon wrote: post #6
heipei wrote: post #3

Earlier quoted context omitted.

I've given up on expecting sane defaults from every piece of software. Some packages work perfectly fine out of the box, or simply work not at peak performance if not tuned slightly. Other software has so many dangerous default settings that it's hard to understand the rationale. Case in point for me is Docker itself. By default it will write logs to json-file and not truncate or rotate these logs. Packages distribut…

Could you point to some of the sane configuration for docker? We are planning to run some of these in production.

This is really the million dollar question. Right now I'm not aware of any single cookbook example of how to tune your server for an optimal docker load. It's all buried inside engineering organizations, or blog posts like the one here.

One of the things the MySQL developers did originally was to ship the code with three examples of the my.cnf config file for small, medium and large memory systems. I wish there was something like that for docker container density and OS tuning parameters.

I think what we need is basically a matrix of OS settings which will max out your docker density for N CPUs and M gigs of memory. There are also probably settings for network configs which depend on your link capacity also.

I can dream of a day when you can spin up an Ubuntu Docker-flavor server which optimizes itself to give the highest density of containers given the hardware (or VM) it's running on.

Re: From 30 to 230 Docker containers per host

#32

Post by Ao7bei3s »

ktpsns wrote: post #9
myrandomcomment wrote: post #7

So at one point we where doing scale testing for our product where we needed to simulate systems running our software connected back to a central point. The idea was to run as many docker containers as we could on a server with 2x24 core and 512GB of RAM. The RAM needed for each container was very small. No matter what the system would start to break around ~1000 containers (this was 4 years ago). After doing may hou…

512GB RAM/2500 containers is still 500MB per container. In former days™ this was enough for a computer to run a complete desktop environment with a web browser and 20 tabs open (source: I had a PC with physically 500MB RAM). Is this really the limit for such a decent equipped machine? (I guess a server grade 48 cores, 512GB RAM should be less then 5kEUR nowadays)

He said "The RAM needed for each container was very small" - the RAM is _not_ the limit here. The point is that running many containers is a very different (hard to compare directly) type of load than the main, well optimized use case of running a single large desktop environment.

Networking in particular: with thousands of containers, now there are lots more interfaces, routes, conntrack entries, "background" traffic, iptables rules etc.

Some problems are algorithmic: for example historically a lot of code has been written with the assumption that the number of network interfaces is small. With >1k interfaces, suddenly O(n) lookups take time. Similarly, iptables rules are run sequentially. etc.

Some resources have limits. Exceeding these may impact performance. In his case the load blew up the ARP cache. Nice!

Re: From 30 to 230 Docker containers per host

#33

Post by cpuguy83 »

heipei wrote: post #3
ec109685 wrote: post #2

Is there any talk of increasing these defaults in higher memory systems. The low defaults feel like foot guns that people stumble into rather than something needed for optimal performance.

I've given up on expecting sane defaults from every piece of software. Some packages work perfectly fine out of the box, or simply work not at peak performance if not tuned slightly. Other software has so many dangerous default settings that it's hard to understand the rationale. Case in point for me is Docker itself. By default it will write logs to json-file and not truncate or rotate these logs. Packages distribut…

Would love to change this default but doing so will end up breaking Kubernetes users at the very least

Maybe we could change the default based on api version.

Re: From 30 to 230 Docker containers per host

#34

Post by myrandomcomment »

Ao7bei3s wrote: post #26
myrandomcomment wrote: post #7

So at one point we where doing scale testing for our product where we needed to simulate systems running our software connected back to a central point. The idea was to run as many docker containers as we could on a server with 2x24 core and 512GB of RAM. The RAM needed for each container was very small. No matter what the system would start to break around ~1000 containers (this was 4 years ago). After doing may hou…

Nice! But how do you tell that from strace? Wouldn't it just show the guests stuck in connect()? With perf, sure, but strace?

It’s has been awhile, but IIRC we saw the net process for ARP being called and getting stuck.

Re: From 30 to 230 Docker containers per host

#35

Post by u6eexrtxjjxjexr »

philsnow wrote: post #30
codegladiator wrote: post #21

Earlier quoted context omitted.

wait, i had windows xp with firefox and all at 256 mb ram ! which i got later upgraded to 384 mbs because i reused another 128 mb ram chip :)

In ~2003, I replaced my desktop that had been Intel-based with a Duron 800MHz system, only I didn't have enough budget to get it the RAM it required (new/different slot iirc), so I only had the 128MB it came with (whereas my old machine had 768MB cobbled together from like six dimms). I figured that one hop over 100Mbit Ethernet to remote memory was going to be faster then swapping to spinning rust (remember this was…

So you're the guy who actually managed to "download more RAM". Congrats!

On a more serious note, gonna add that trick to my book - still plenty of rust spinnin' round

Re: From 30 to 230 Docker containers per host

#39

Post by viraptor »

ktpsns wrote: post #9
myrandomcomment wrote: post #7

So at one point we where doing scale testing for our product where we needed to simulate systems running our software connected back to a central point. The idea was to run as many docker containers as we could on a server with 2x24 core and 512GB of RAM. The RAM needed for each container was very small. No matter what the system would start to break around ~1000 containers (this was 4 years ago). After doing may hou…

512GB RAM/2500 containers is still 500MB per container. In former days™ this was enough for a computer to run a complete desktop environment with a web browser and 20 tabs open (source: I had a PC with physically 500MB RAM). Is this really the limit for such a decent equipped machine? (I guess a server grade 48 cores, 512GB RAM should be less then 5kEUR nowadays)

You're assuming that all the memory was used, even though it's not specified in the post. 512GB/2500 is the upper limit. Anything between 0 and that could be the case.

Re: From 30 to 230 Docker containers per host

#40

Post by mkl »

_nickwhite wrote: post #37

"With /proc/sys/kernel/pid_max defaulting to 32768 we actually ran out of PIDs. We increased that limit vastly, probably way beyond what we currently need, to 500000. Actuall limit on 64bit systems is 222" Time to start thinking about 128bit systems!

Copy-paste error: it's 2^22, which is 4194304.
Post reply on HN