Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker and the PID 1 zombie reaping problem

blog.phusion.nl

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Docker and the PID 1 zombie reaping problem

#51

Post by m00dy »

ekimekim wrote: post #42
m00dy wrote: post #30

>“As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes.” To the author-> Do you know how many processes a kernel can handle ? root@lisa2:~# cat /proc/sys/kernel/pid_max 32768 fyi.

Suppose a service has a timed routine that spawns a child every 10 seconds. Unfortunately, this child has a bug and starts leaking zombies to PID1. Congratulations, you now have approx 9 hours until your entire system stops being able to spawn new processes.

You could have a nuclear war near the datacenter where you host that broken software & container. I think it has a better probability to exist in a real world.

Re: Docker and the PID 1 zombie reaping problem

#52

Post by masklinn »

emeraldd wrote: post #34
masklinn wrote: post #21

Earlier quoted context omitted.

Supervisord is not an init and must not be run as pid1, that's explicitly mentioned in its documentation[0]: > It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. There are micro-inits you could use instead though, s6 should be runnable as pid1 for instance. [0] http://supervisord.org/in…

I'm a little confused here. Especially considering that the docker site lists supervisord explicitly with instructions on how to use it .... see: https://docs.docker.com/articles/using_supervisord/

There's no problem with running supervisord, inside or outside of docker, there's a problem running supervisord as an init (pid1). You can run openrc or systemd or s6 or runit or minit or whatever and use supervisord on top of that, you just shouldn't use supervisord directly as pid1.

Re: Docker and the PID 1 zombie reaping problem

#53

Post by shykes »

edwintorok wrote: post #48
shykes wrote: post #44

Hi, Docker author. Just wanted to say that we are aware of the problem, understand the underlying mechanics, and plan to fix it. In the meantime application-level fixes like this can definitely help. One slight difficulty is that some applications do want to be pid1. While 99% of the ecosystem wants the problem of being pid1 to be handled natively by Docker, the remaining 1% (people who write init systems, or platfor…

As long as you provide a choice on what to run as PID1, and that init system immediately/automatically runs the ENTRYPOINT / CMD / run command I think it is fine to change the behaviour to the correct one. Please don't make the mistake to hardcode PID1, and worst of all not to systemd ... as systemd causes enough trouble in docker containers already (try debian host, fedora:20 docker and upgrade if you can).

Don't worry, we believe in small composable tools, for example we are planning to break up Docker into more discrete binaries. So we're not about to bundle systemd into every Docker installation ;)

Re: Docker and the PID 1 zombie reaping problem

#54

Post by Florin_Andrei »

FooBarWidget wrote: post #49
shykes wrote: post #44

Hi, Docker author. Just wanted to say that we are aware of the problem, understand the underlying mechanics, and plan to fix it. In the meantime application-level fixes like this can definitely help. One slight difficulty is that some applications do want to be pid1. While 99% of the ecosystem wants the problem of being pid1 to be handled natively by Docker, the remaining 1% (people who write init systems, or platfor…

Having a config option is a good idea. I vote for changing the default behavior because Docker has so much growth potential left. Leaving the default as-is is problematic for the majority of people, both current and future. I think that breaking it is acceptable because a lot of the people who use Docker in production -- especially those that are expecting to be PID 1 -- are early adopters and will forgive you for th…

Yes, change it now, before it's everywhere. The longer you wait, the worse the problem becomes.

Re: Docker and the PID 1 zombie reaping problem

#55

Post by detaro »

sandGorgon wrote: post #28

I'm not sure writing yet another pid 1 tool is such a good idea. I have been using Docker in production for a few months now and I use supervisord as pid 1. I am able to hook it to most app servers (python,rails,etc) as well as rsyslogd . logging is a big pain on Docker, and you should think of your logging strategy before the pid 1. How are you rotating your logs and is your pid 1 restarting your app on failure? For…

Supervisord docs say explicitly that it is not meant be used as PID 1. Does it still perform all the necessary functions?

Re: Docker and the PID 1 zombie reaping problem

#56

Post by kstenerud »

mborch wrote: post #23
FooBarWidget wrote: post #8

Earlier quoted context omitted.

Actually, the Python process eats less than 5 MB of RAM. Most other stuff on the system eats more RAM. Writing my_init took much more time than you think, and I do not regret choosing Python. My_init not only handles process reaping, but also environment variable handling and bunch of other stuff. Writing text parsers in C is a huge pain (libc has no regexps that are as easy to use as Python). I do not oppose rewriti…

Fair enough, but for the record it's at least 10 MB (try starting up Python and import your list of modules). The environment variable handling and other stuff could perhaps be handled in a separate bash-script (or Python), such that the scope of the actual reaper is sufficiently small (to use a lower-level language).

5-10MB is a big nothing, unless you're talking about running hundreds of images that consume 20-30mb each. in which case you would have some custom image structure to minimize everything anyway. But then you'd want to consider the infrastructure development cost vs just buying more iron.

There comes a point where you have to take a look at the diminishing returns and say "That's good enough."

Re: Docker and the PID 1 zombie reaping problem

#57

Post by KaiserPro »

bitwize wrote: post #41
KaiserPro wrote: post #29

I know this will cost me a lot of karma, but surely this is the point that you say, lets just use a hypervisor? yes, I know its much easier to just use docker, but at the point that you have to write a new INIT, doesn't that strike you as time you used something thats not easy anymore? Things like "make sure we don't loose syslog" make me shudder. Yes supposedly there is an argument that docker is faster. However as…

This is actually why we all need to support systemd. It is already an init system, and a sophisticated one; now that it has integrated container support it's only a matter of time before systemd's pid 1 assumes zombie reaping duties for the host and all container images. People who criticize systemd fail to realize that it provides solutoons for many problems under Linux, and even problems with the Unix model itself,…

Yes, its provides a supervisord type mechanism, thats great, so is dependencies but outweighed by the following:

The horrid shite that is the journald. whoever thought it was a brilliant idea to smoosh grep tail and vi together is someone who's never had to debug anything hung over.

seriously why in living fuck do I have to use your shitty tool to look at system logs? The whole point of linux that you are supposed to be able to use anything to open anything. Sure its a stupid idea to use most tools, but thats not your decision.

Seriously do I have to learn a completely new syntax to find all things that happened at 10:23:04 three days ago? Everywhere else I can use vim/nano/emacs/awk/grep/sed.

but here? no I'm to stupid to understand those tools, here use this one. We've thought of every use case, if you're having trouble its because you're either a hater, idiot, sysadmin, or too old.

you can tell that the people who write this shit are not the poor fuckers who have to tune/support/debug/migrate or use it.

So no, I'm not going to "get behind systemd" until journal is fit for purpose.

Re: Docker and the PID 1 zombie reaping problem

#58

Post by KaiserPro »

xyzzy_plugh wrote: post #50
TheSwordsman wrote: post #45

Earlier quoted context omitted.

And people who blindly push systemd fail to realize that we've alreqady solved these problems. A more complex system doesn't solve it. So instead of a 350 line Python script, you think systemd is better? I disagree.

A more complex system? How is solving this problem universally a bad thing? Is having _one_ way to do things that works everywhere so bad? Having a container manager do things I wouldn't expect, like have issues with zombies or dropping syslog messages, makes me never want to touch it.

My dear chap, one way does not work everywhere. if that were the case we'd all have agreed to just use vim, or emacs, or sublime, or nedit, or nano, or ed......

Re: Docker and the PID 1 zombie reaping problem

#59

Post by KaiserPro »

shykes wrote: post #46
koffiezet wrote: post #39

Earlier quoted context omitted.

Docker does different things than VM's, but they work together perfectly. I run my all my Docker hosts on a VMWare infrastructure, which gives me best of both worlds. Yes - for some things, VMWare can't be beaten - but that's the nice thing: they are not mutually exclusive. Both are very handy tools in my toolbox, and neither are a one-stop solution. Docker sure adds complexity - but also a huge amount of flexibility…

> My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications. You should take a look at https://github.com/docker/machine and https://github.com/docker/swarm , they were designed with exactly that goal in mind!

Sadly, it still lags behind even KVM, it can't hot migrate, which means zero downtime maintenance is impossible, along with transparent HA.

Re: Docker and the PID 1 zombie reaping problem

#60

Post by FooBarWidget »

edwintorok wrote: post #27
FooBarWidget wrote: post #8

Earlier quoted context omitted.

Actually, the Python process eats less than 5 MB of RAM. Most other stuff on the system eats more RAM. Writing my_init took much more time than you think, and I do not regret choosing Python. My_init not only handles process reaping, but also environment variable handling and bunch of other stuff. Writing text parsers in C is a huge pain (libc has no regexps that are as easy to use as Python). I do not oppose rewriti…

Why not use already existing init? runit seems small enough: http://smarden.org/runit/ Or if you want small have a look here for minimal PID 1 "So how should init be done right?" http://ewontfix.com/14/

Baseimage-docker does use Runit. my_init calls Runit.
Post reply on HN