Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tokio and Prctl = Nasty Bug

kobzol.github.io

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Tokio and Prctl = Nasty Bug

#41

Post by fc417fc802 »

kobzol wrote: post #35
deathanatos wrote: post #22

> In particular, it is not always possible for HQ to ensure that when a process that spawns tasks (called worker) quits unexpectedly (e.g. when it receives SIGKILL), its spawned tasks will be cleaned up. Sadly, Linux does not seem to provide any way of implementing perfect structured process management in user space. In other words, when a parent process dies, it is possible for its (grand)children to continue execut…

Yeah by user space I just meant without root, sorry. HQ runs on supercomputers where the environment is heavily locked up, even Docker doesn't work. I think that PID namespaces aren't really possible, but I haven't tried it yet. Subreaper doesn't help, because if the worker dies, the children aren't killed, even if they are the children of the worker, they will be just reparented to init.

> I think that PID namespaces aren't really possible

Depends on the cluster. If they're using nix or guix then they presumably enabled user namespaces but a few years ago guix had an article about (generally shitty) workarounds for people running in environments where those were disabled.

Edit: Maybe you should have two code paths. A fast namespaced one and the slower old one as a fallback.

Re: Tokio and Prctl = Nasty Bug

#42

Post by timhh »

TheDong wrote: post #4

I think they don't want PR_SET_PDEATHSIG but rather PR_SET_CHILD_SUBREAPER, which I think would be both more correct than PDEATHSIG for letting them wait on grand-children / preventing grand-child-zombies, while also avoiding the issue they ran into here entirely. They would need one special "main thread" that deals with reaping and that isn't subject to tokio's runtime cleaning it up, but presumably they already hav…

> PR_SET_CHILD_SUBREAPER

I wrote a tool that does just this: https://github.com/timmmm/anakin

If you run `anakin ` it will kill any orphan processes that makes.

However is still isn't the true "orphans of this process must automatically die" option that everyone writing job control software wants - if `anakin` itself somehow crashes then the orphans can live again.

Still it was the best I could come up with that didn't need root.

Re: Tokio and Prctl = Nasty Bug

#43

Post by fc417fc802 »

oguz-ismail wrote: post #16
rendaw wrote: post #10

Earlier quoted context omitted.

Are there any differences between threads and processes in how signals are handled? I recently learned that aside from processes there are process groups, process sessions (setsid), process group and session leaders, trees have associated VT ownership data, systemd sessions (which seem to be inherited by the entire subtree and can't be purged), and possibly other layered metadata spaces that I haven't heard of yet. A…

> I really wish there were an overview of all these things and how they interact with eachother somewhere. man 7 signal

Also see `man 2 clone` and `man 7 cgroups`.

Re: Tokio and Prctl = Nasty Bug

#44

Post by TheDong »

kobzol wrote: post #33
eqvinox wrote: post #14

Earlier quoted context omitted.

> PR_SET_PDEATHSIG automatically kills your children if you die, but unfortunately doesn’t extend to their descendants It indirectly does, unless you unset it the child dying will trigger another run of PDEATHSIG on the grandchildren, and so on. (The setting is retained across forks, as shown in the original article.)

It is sadly not propagated to grandchildren. I tries the subreaper approach, but it doesn't help. The children are reparented to the worker, but when the worker dies, they are then just reparented to init, like normally.

You also need to specifically have the subreaper process call the "wait" syscall, and wait for all children, otherwise of course they'll end up reparented to init.

If you want to write a process manager, one of the process manager's responsibilities is waiting on its children.

Re: Tokio and Prctl = Nasty Bug

#45

Post by skissane »

fc417fc802 wrote: post #40
skissane wrote: post #28

Earlier quoted context omitted.

Digressing a little, but Glibc’s pthreads implementation is painful, because they don’t provide any public API to map a pthread_t to the kernel TID, except for the horrendously awful thread_db. Of course, for the current thread, you can just call gettid() - but if you want to map pthread_t to TID for another thread, the thread_db abomination is the only supported way. Bionic supplies a nice simple pthread_gettid_np()…

Huh I never really thought about that before. Seems like a glaring oversight but then again do any POSIX APIs even involve threads? Which itself illustrates the absurdity because what modern OS doesn't support multiple scheduling entities per virtual address space? Or should I have said per thread group? What was a process supposed to be again? (And what was the point of the thing?) Digressing the conversation furthe…

> do any POSIX APIs even involve threads?

pthreads (POSIX threads) is itself a POSIX API

I guess one reason why it doesn’t have any TID concept, is although Linux nowadays uses 1:1 threading (one kernel thread per user-space thread), historically many Unix thread libraries were designed to use 1:N threading (a single kernel thread runs multiple user space threads) or M:N threading (a pool of kernel threads runs a pool of user space threads where the two pools differ in size). Plus, while Linux went with the model that processes and threads are basically two slightly different variants of the same thing, in other POSIX implementations they are completely distinct object types. Since pthreads are designed to support such a wide variety of implementation strategies, they can’t assume threads have any kernel-maintained unique ID, because in some of those implementation strategies there might not be one.

> I notice in the docs that CLONE_SIGHAND requires CLONE_VM

I think this is necessary? If it wasn’t, the child might load new code (dlopen or JIT) and then install a signal handler pointing to it. With CLONE_SIGHAND, it shares signal handler with parent. But without CLONE_VM, the memory mapping containing the new code wouldn’t exist in the parent, meaning instant segfault as soon as the signal is delivered

> and CLONE_THREAD requires CLONE_SIGHAND. Any idea if there's a technical reason for that or is it just POSIX constraints needlessly infecting the kernel?

Well, this one is more POSIX (and historical Unix before it). Signals are primarily a process-level construct in Unix/POSIX, not thread-level – since back when signals were invented, threads hadn’t been invented yet (on Unix–PL/I running under OS/360 MVT already had multithreading, which it called 'multitasking', in 1968, Unix development didn't start until 1969). Although we’ve now got per-thread signal masks and thread-directed signals, the actual handlers are still per-process

Also, I think another reason for disallowing certain combination of clone() flags: obscure combinations can expose bugs, possibly even security vulnerabilities; if there is no great demand for a specific combination, the kernel devs may conclude it is safest to disallow it

Re: Tokio and Prctl = Nasty Bug

#46

Post by fc417fc802 »

> I don’t know how to tell it to only send the signal when the parent process (not thread) dies

What about an `atexit` handler and maintaining a table of child processes to kill? If you need something more robust in the face of adverse termination you could instead spawn an independent process, call `wait` on the primary process, and then handle any remaining cleanup.

Re: Tokio and Prctl = Nasty Bug

#47

Post by badmintonbaseba »

fc417fc802 wrote: post #46

> I don’t know how to tell it to only send the signal when the parent process (not thread) dies What about an `atexit` handler and maintaining a table of child processes to kill? If you need something more robust in the face of adverse termination you could instead spawn an independent process, call `wait` on the primary process, and then handle any remaining cleanup.

Presumably the author would like something that's handled by the kernel and not user space, so even a SIGKILL-ed parent process would trigger the reaping of the children.

Re: Tokio and Prctl = Nasty Bug

#48

Post by fc417fc802 »

badmintonbaseba wrote: post #47
fc417fc802 wrote: post #46

> I don’t know how to tell it to only send the signal when the parent process (not thread) dies What about an `atexit` handler and maintaining a table of child processes to kill? If you need something more robust in the face of adverse termination you could instead spawn an independent process, call `wait` on the primary process, and then handle any remaining cleanup.

Presumably the author would like something that's handled by the kernel and not user space, so even a SIGKILL-ed parent process would trigger the reaping of the children.

That would be a pid namespace but was passed over for compatibility reasons I guess.

Addressing a SIGKILL'd parent specifically, what about daemonizing the cleanup process but instead of fork use clone with the CLONE_VM flag?

Re: Tokio and Prctl = Nasty Bug

#49

Post by emmelaich »

Seeing the first mention of 10 seconds, I thought (jokingly) - Why not grep the source for the 10 second value.

Jokingly, because I thought it would be an emergent property, not a literal value.

Turns out it was a literal value after all and grepping would have helped!

Re: Tokio and Prctl = Nasty Bug

#50

Post by badmintonbaseba »

timhh wrote: post #42
TheDong wrote: post #4

I think they don't want PR_SET_PDEATHSIG but rather PR_SET_CHILD_SUBREAPER, which I think would be both more correct than PDEATHSIG for letting them wait on grand-children / preventing grand-child-zombies, while also avoiding the issue they ran into here entirely. They would need one special "main thread" that deals with reaping and that isn't subject to tokio's runtime cleaning it up, but presumably they already hav…

> PR_SET_CHILD_SUBREAPER I wrote a tool that does just this: https://github.com/timmmm/anakin If you run `anakin ` it will kill any orphan processes that makes. However is still isn't the true "orphans of this process must automatically die" option that everyone writing job control software wants - if `anakin` itself somehow crashes then the orphans can live again. Still it was the best I could come up with that didn…

The name of the tool is on point.
Post reply on HN