Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systemd vs. Docker

lwn.net

Post reply on HN
Print view View original

71–80 of 97 posts

Re: Systemd vs. Docker

#71

Post by atemerev »

qwertyuiop924 wrote: post #57
vidarh wrote: post #38

Earlier quoted context omitted.

Isolation and integration are not opposites. You can increase isolation (through e.g. judicious application of cgroups, which systemd encourages and makes use of) while also increasing integration (e.g. sharing APIs). Docker is to me far more problematic when it comes to integration. It is trying to make everything go through itself, without providing a fraction of the scheduling and management capabilities that syst…

Integration is all well and good, I suppose. Until you put all your integration into a single process, which will kill your system if it ever crashes. Then I will have some words to say about you. Especially if you ever, ever, EVER, EVER, EVER, use assert, in ANY situation you could ever potentially recover from.

Why recover if restarting is much simpler? Like in Erlang and Akka, this leads to incredibly reliable systems.

Re: Systemd vs. Docker

#72

Post by masklinn »

js2 wrote: post #69

One of these is killing "zombie" processes that have been abandoned by their calling session. That's funny terminolgy, isn't it? Killing a process usually means sending it a signal, typically TERM or KILL, that causes it to exit. But a zombie process is one that has already exited, but hasn't been waited for by its parent, where its parent is either the process that spawned it, or if that process has died, the proces…

No, it's a zombie in the normal sense, the killing here is not sending it a signal but reaping zombie processes (in the sense of personified death reaping souls) by waiting on it.

Things would probably be clearer if the quotes were around "killing" rather than "zombie", mayhaps the interviewer/writer was unfamiliar with the terminology.

Re: Systemd vs. Docker

#73

Post by masklinn »

pas wrote: post #50
vidarh wrote: post #33

Earlier quoted context omitted.

It has been a reasonably big issue. E.g. I kept seeing zombies with Consul for a while until we realised that every single Consul Docker container on Dockerhub just had Consul run as pid 1 in the container (this is a while ago, no idea if that's still the case), without realising that Consul health checks then could end up as zombies if you weren't very careful about how you wrote them (e.g. typical example: Spawning…

Seeing how even the trivial pid1 "scripts" solve the problem, it's truly baffling why Docker doesn't have a --with-reaper flag. Also thanks for the Consul example, makes it much-much easier to see the issue and argue for a general solution. (So not every random app/project/service/daemon has to implement pid1 functionality.)

> Seeing how even the trivial pid1 "scripts" solve the problem, it's truly baffling why Docker doesn't have a --with-reaper flag.

That doesn't fix the issue since you need to know about the issue and accept that it exists, at that point you can just as easily use one of the micro-inits available.

The alternative is to enable it by default, but now you've broken BC for the weirdo who actually expects orphan processes to be adopted by the root process they're starting.

Re: Systemd vs. Docker

#74

Post by cpuguy83 »

masklinn wrote: post #18
thwarted wrote: post #13

Poettering says that PID 1 has special requirements. One of these is killing "zombie" processes that have been abandoned by their calling session. This is a real problem for Docker since the application runs as PID 1 and does not handle the zombie processes. For example, containers running the Oracle database can end up with thousands of zombie processes. Why does Poettering keep claiming this when he's the one who s…

That doesn't have anything to do with Poettering's quote. PR_SET_CHILD_SUBREAPER moves the ownership of an orphaned process to whichever process was selected rather than the default PID1, and that only works for descendant of the subreaper. The problem pointed by the quote is that normal software doesn't go around checking if it has zombie children and waiting on them, so in a container with random software S set as…

This is true, but what if the thing that spins up the actual container process sets this?

Re: Systemd vs. Docker

#75

Post by the_why_of_y »

qwertyuiop924 wrote: post #57
vidarh wrote: post #38

Earlier quoted context omitted.

Isolation and integration are not opposites. You can increase isolation (through e.g. judicious application of cgroups, which systemd encourages and makes use of) while also increasing integration (e.g. sharing APIs). Docker is to me far more problematic when it comes to integration. It is trying to make everything go through itself, without providing a fraction of the scheduling and management capabilities that syst…

Integration is all well and good, I suppose. Until you put all your integration into a single process, which will kill your system if it ever crashes. Then I will have some words to say about you. Especially if you ever, ever, EVER, EVER, EVER, use assert, in ANY situation you could ever potentially recover from.

I disagree with your position on assertions and much prefer Bryan Cantrill's, because it results in bugs actually getting fixed:

  Hope is not a strategy, including for your
  software.  If your state has  become corrupt,
  it is incumbent upon you to die and donate
  your body to science, where it can be
  debugged.

Re: Systemd vs. Docker

#76

Post by masklinn »

cpuguy83 wrote: post #74
masklinn wrote: post #18

Earlier quoted context omitted.

That doesn't have anything to do with Poettering's quote. PR_SET_CHILD_SUBREAPER moves the ownership of an orphaned process to whichever process was selected rather than the default PID1, and that only works for descendant of the subreaper. The problem pointed by the quote is that normal software doesn't go around checking if it has zombie children and waiting on them, so in a container with random software S set as…

This is true, but what if the thing that spins up the actual container process sets this?

What do you mean "the thing that spins up the actual container"? The root process for the container? It's already PID1. The external process creating the container? It's sitting outside the container and "below" PID1, what could that do that'd be of any use?

Re: Systemd vs. Docker

#77

Post by shykes »

masklinn wrote: post #73
pas wrote: post #50

Earlier quoted context omitted.

Seeing how even the trivial pid1 "scripts" solve the problem, it's truly baffling why Docker doesn't have a --with-reaper flag. Also thanks for the Consul example, makes it much-much easier to see the issue and argue for a general solution. (So not every random app/project/service/daemon has to implement pid1 functionality.)

> Seeing how even the trivial pid1 "scripts" solve the problem, it's truly baffling why Docker doesn't have a --with-reaper flag. That doesn't fix the issue since you need to know about the issue and accept that it exists, at that point you can just as easily use one of the micro-inits available. The alternative is to enable it by default, but now you've broken BC for the weirdo who actually expects orphan processes…

Yes, the problem is that we would need to change the default behavior of Docker, which many people and scripts expect to be stable. It's a case of interface stability vs. functionality improvement. So far interface stability has won. I personnally think it would be better to change the default, but anything that breaks an interface, even a subtle implicit one, has the burden of arguing a solution, thinking through migration issues, submitting patches... So far I have seen a lot of drive-by criticisms and dismissal of the need to even discuss the tradeoff (see for example this lovely fellow: https://lwn.net/Articles/677419/). But I have not seen anyone stepping up to do the work.

We all pick our battles - including me of course!

Re: Systemd vs. Docker

#78

Post by thwarted »

masklinn wrote: post #18
thwarted wrote: post #13

Poettering says that PID 1 has special requirements. One of these is killing "zombie" processes that have been abandoned by their calling session. This is a real problem for Docker since the application runs as PID 1 and does not handle the zombie processes. For example, containers running the Oracle database can end up with thousands of zombie processes. Why does Poettering keep claiming this when he's the one who s…

That doesn't have anything to do with Poettering's quote. PR_SET_CHILD_SUBREAPER moves the ownership of an orphaned process to whichever process was selected rather than the default PID1, and that only works for descendant of the subreaper. The problem pointed by the quote is that normal software doesn't go around checking if it has zombie children and waiting on them, so in a container with random software S set as…

Yes it does. He's claiming that systemd should manage the container processes as pid1, because systemd will then clean up the zombies. But anything that reaps zombies can be pid1 -- systemd isn't special in this regard. And even if you did use something that didn't reap zombies as pid1, you could leverage PR_SET_CHILD_SUBREAPER as some other non-pid1 process to grab zombies for descendants it spawns.

If you do use PR_SET_CHILD_SUBREAPER, then you need to reap whatever gets reparented to you; if you don't do this then the process table will eventually fill up with zombies. He is correct that few programs do that, but there's nothing that requires that to be done by pid1 if all the processes within the container are spawned by something that provides that functionality and uses PR_SET_CHILD_SUBREAPER.

Re: Systemd vs. Docker

#79

Post by storrgie »

why not systemd-nspawn (zoidberg voice)

Seems like the way Fedora is packaging systemd for 24 is going to move systemd-nspawn to a level of maturity that will likely surpass some of the clunky issues folks have with running docker.

Re: Systemd vs. Docker

#80

Post by ChrisArgyle »

js2 wrote: post #69

One of these is killing "zombie" processes that have been abandoned by their calling session. That's funny terminolgy, isn't it? Killing a process usually means sending it a signal, typically TERM or KILL, that causes it to exit. But a zombie process is one that has already exited, but hasn't been waited for by its parent, where its parent is either the process that spawned it, or if that process has died, the proces…

If we're being technical the author should have wrote "reaping" not "killing". It's a very different process.

The use of quotes is probably an acknowledgement that the term "zombie" is not universal. For example Linux uses "defunct" instead.

Basically, zombie processes happen when a child process exits but the parent process--the one that spawned it--doesn't reap it. [1]

[1] https://en.wikipedia.org/wiki/Zombie_process

Post reply on HN