31–40 of 72 posts
>“As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes.” To the author-> Do you know how many processes a kernel can handle ? root@lisa2:~# cat /proc/sys/kernel/pid_max 32768 fyi.
Do you know how many values a 32-bit integer can represent? Surely no one ever has to worry about overflows?
If you use Docker containers as a form of lightweight immutable infrastructure you won't care about zombie processes because you won't be restarting stopped containers. Instead you just `docker run` every time you want to run a new container. The example of a corrupt file is moot because your containers shouldn't be writing to the file system with the intention of that file being around for long. This also applies to databases, which I'm not fond of running in containers at all (setting aside highly distributed databases, perhaps).
>“As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes.” To the author-> Do you know how many processes a kernel can handle ? root@lisa2:~# cat /proc/sys/kernel/pid_max 32768 fyi.
# echo 1000000 >/proc/sys/kernel/pid_max
# cat /proc/sys/kernel/pid_max
1000000
Why write your own mini-init instead of using something like supervisord?
Supervisord is not an init and must not be run as pid1, that's explicitly mentioned in its documentation[0]: > It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. There are micro-inits you could use instead though, s6 should be runnable as pid1 for instance. [0] http://supervisord.org/in…
I know this will cost me a lot of karma, but surely this is the point that you say, lets just use a hypervisor? yes, I know its much easier to just use docker, but at the point that you have to write a new INIT, doesn't that strike you as time you used something thats not easy anymore? Things like "make sure we don't loose syslog" make me shudder. Yes supposedly there is an argument that docker is faster. However as…
I know this will cost me a lot of karma, but surely this is the point that you say, lets just use a hypervisor? yes, I know its much easier to just use docker, but at the point that you have to write a new INIT, doesn't that strike you as time you used something thats not easy anymore? Things like "make sure we don't loose syslog" make me shudder. Yes supposedly there is an argument that docker is faster. However as…
You could literally use bash as the init process if you do not care about shutting down cleanly (or are willing to have a more complicated shutdown procedure that 'docker stop').
More seriously, the issues raised in this article are real, and I do believe the author has done a fine job breaking down the UNIX process hierarchy and defined responsibilities.
I'd say it's a shame as this is yet one more important stability/scalability/security/performance item the Docker team has not thought of (or maybe their busy writing more eco-system apps), but I do believe this is a container problem at large seeing that the official App Container Spec has not addresses it.
For those criticizing the need for "yet another pid 1", well, there doesn't seem to be any way around it that can guarantee the same outcome. Yes, it's understandable having reservations about Python being the tool of choice for such an init system, and perhaps C would be a better choice allowing more fine-grained control and guarantees. My guess is the article author was most familiar with Python and wanted to have a POC for the write-up, which is satisfactory.
A simpler and better solution for this problem is: do not let apps run in background (double-fork/detach) in containers. Most applications that background by default can override this with a command-line flag. And certainly don't run programs that execute other programs that background. Keep it simple, and try to limit yourself to 1 process per container. Sure there might be situations where this cannot be avoided, but in 99% of the situations - this is not a problem. Yes you have to be aware of this potential problem, but I am currently running tons of containers, of which only a few run supervisord (which handles reaping too) to run multiple processes within one container because it was the sane thing to do. Stale zombie processes on my docker hosts? None. These precautions in my opinion are only required for badly behaving software. And yes - there is software like that.
I know this will cost me a lot of karma, but surely this is the point that you say, lets just use a hypervisor? yes, I know its much easier to just use docker, but at the point that you have to write a new INIT, doesn't that strike you as time you used something thats not easy anymore? Things like "make sure we don't loose syslog" make me shudder. Yes supposedly there is an argument that docker is faster. However as…
And then:
1) The problem is hugely exaggerated. It only applies to badly written software. And make sure we don't lose syslog? Seriously? just mount /dev/log into your container (it's a unix socket after all) and you're done.
2) Arguably faster? Try booting up 15 instances of app X. With docker, that's achieved in less than a second. On VMWare this has a huge memory and CPU overhead, not to mention boot-up times.
3) Try versioning VM's. Good luck. I have 100% reproducible, versioned images generated and tested in our CI system, that I can easily deploy in a VM on my laptop where I will have 100% the same setup as it will in test/qa/prod. To deploy the new version, start new instance, point load balancer/proxy/whatever to the new instance (happens automagically btw). Everything goes well? Stop old instance. Something goes wrong? Point proxy to old instance, and kill the badly-behaving. If everything went well: zero downtime. If you detect something goes wrong, it's nearly instant to switch back to the previous version. Also, testing this is a lot easier than with VM's. Sure you could do that with VM's too - but the overhead is massive, and takes a lot more time.
My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications.
I think needing to run an init system with Docker is a solution to a problem you've created for yourself. From reading this post and other posts on your blog about Docker it sounds like you are `docker start`-ing containers that have been `docker stop`-ped. In my experience, using `docker start` isn't a best practice and I design my containers in a way that I won't need to restart a stopped container. If you use Dock…
As an example, suppose we have a simple web server that executes CGI scripts written in bash (I know this is kinda contrived but it illustrates my point). I do a request to the server, it runs a bash script. The script runs grep. While grep is executing, the web server decides bash has taken too long and SIGKILLs it. Then grep is adopted by your PID1, in this case your webserver. But the webserver doesn't know its there, and so never calls wait(). You now have a zombie process that will last until you stop the container.