Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tokio and Prctl = Nasty Bug

kobzol.github.io

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Tokio and Prctl = Nasty Bug

#31

Post by kobzol »

cryptonector wrote: post #23

If you want to be able to spawn processes that fask then `fork()` is NOT your friend. You want either `vfork()` (or `clone()` equivalent) or `posix_spawn()`. `fork()` is inherently very slow due to the need to either copy the VM of the parent, or arrange to copy pages on write, or copy the resident set of the parent (then copy any pages paged-in when those page-in events happen -- all three of these options are very…

The stdlib already mostly does all of that :)

Check out https://kobzol.github.io/rust/2024/01/28/process-spawning-pe....

Re: Tokio and Prctl = Nasty Bug

#32

Post by kobzol »

bhawks wrote: post #13

Normally I'd stay away from job control posix APIs - but since HyperQueue is a job control system, it might be appropriate if the worker was a session leader. If it dies than all its subprocesses would receive SIGHUP - which is fatal by default. Generally you'd use this functionality to implement something like sshd or an interactive shell. HQ seems roughly analogous. https://notes.shichao.io/apue/ch9/#sessions

I do use setsid when spawning the children (I omitted it from the post, but I set it in the dsme pre_exec call where I configure DEATHSIG) but they don't receive any signal, IIRC. Or if they do, it does not seem to be propagated to their children.

Re: Tokio and Prctl = Nasty Bug

#33

Post by kobzol »

eqvinox wrote: post #14
skissane wrote: post #11

Earlier quoted context omitted.

> I think they don't want PR_SET_PDEATHSIG but rather PR_SET_CHILD_SUBREAPER, which I think would be both more correct than PDEATHSIG for letting them wait on grand-children / preventing grand-child-zombies, while also avoiding the issue they ran into here entirely. PR_SET_PDEATHSIG automatically kills your children if you die, but unfortunately doesn’t extend to their descendants As far as I’m aware, PR_SET_CHILD_SU…

> PR_SET_PDEATHSIG automatically kills your children if you die, but unfortunately doesn’t extend to their descendants It indirectly does, unless you unset it the child dying will trigger another run of PDEATHSIG on the grandchildren, and so on. (The setting is retained across forks, as shown in the original article.)

It is sadly not propagated to grandchildren.

I tries the subreaper approach, but it doesn't help. The children are reparented to the worker, but when the worker dies, they are then just reparented to init, like normally.

Re: Tokio and Prctl = Nasty Bug

#34

Post by kobzol »

vlovich123 wrote: post #5

> Edit: Someone on Reddit sent me a link to a method that can override the thread keep-alive duration. Its description makes it clear why the tasks were failing after exactly 10 seconds > Yeah, testing if a task can run for 20 seconds isn’t great, but hey, at least it’s something Well a reasonable thing to me is then to use the override within the test to shorten it (e.g. to 1s & use a 2s timeout).

Could be done, yeah, but 20s isn't that much, and I'd like to avoid adding more test-only magic environment variables zo configure this (our end-to-end tests are in Python and they use HQ as a binary).

Re: Tokio and Prctl = Nasty Bug

#35

Post by kobzol »

deathanatos wrote: post #22

> In particular, it is not always possible for HQ to ensure that when a process that spawns tasks (called worker) quits unexpectedly (e.g. when it receives SIGKILL), its spawned tasks will be cleaned up. Sadly, Linux does not seem to provide any way of implementing perfect structured process management in user space. In other words, when a parent process dies, it is possible for its (grand)children to continue execut…

Yeah by user space I just meant without root, sorry. HQ runs on supercomputers where the environment is heavily locked up, even Docker doesn't work. I think that PID namespaces aren't really possible, but I haven't tried it yet.

Subreaper doesn't help, because if the worker dies, the children aren't killed, even if they are the children of the worker, they will be just reparented to init.

Re: Tokio and Prctl = Nasty Bug

#37

Post by skissane »

zokier wrote: post #27
skissane wrote: post #20

Earlier quoted context omitted.

> The setting is retained across forks, as shown in the original article That’s not what the man page says: > The parent-death signal setting is cleared for the child of a fork(2). https://man7.org/linux/man-pages/man2/pr_set_pdeathsig.2cons... Unless the man page is wrong?

I wonder if this is difference between libc fork (which calls clone syscall) and kernel fork syscall.

No, it isn’t. Neither glibc fork nor kernel fork syscall provide any special handling for PDEATHSIG beyond what clone syscall does.

Re: Tokio and Prctl = Nasty Bug

#38

Post by IshKebab »

> Sadly, Linux does not seem to provide any way of implementing perfect structured process management in user space. In other words, when a parent process dies, it is possible for its (grand)children to continue executing. There is a solution for this called PID namespaces, but it requires elevated privileges, and also seems a bit too heavyweight for HyperQueue.

Yeah Linux process management is a bit of a shit show. I didn't know about this sigdeath thing though. That sounds maybe useful. Is it transitive though?

Re: Tokio and Prctl = Nasty Bug

#39

Post by rendaw »

skissane wrote: post #18
rendaw wrote: post #10

Earlier quoted context omitted.

Are there any differences between threads and processes in how signals are handled? I recently learned that aside from processes there are process groups, process sessions (setsid), process group and session leaders, trees have associated VT ownership data, systemd sessions (which seem to be inherited by the entire subtree and can't be purged), and possibly other layered metadata spaces that I haven't heard of yet. A…

> Are there any differences between threads and processes in how signals are handled? Yes. As signal(7) notes [0], Linux has both “process-directed signals” (which can be handled by any thread in a process), and “thread-directed signals” (which are targeted at a specific thread and only handled by that thread). For user-generated signals, the classification depends on which syscall you use (kill/rt_sigqueueinfo gener…

Oh thanks! I was recently having `runuser -l` silently not do the session setup because of the systemd thing, so maybe there's a better way (than laundering it through a process launcher daemon in a separate tree) to handle that.

I forgot capabilities with another 5 layers (+) of different flags and applied differently to processes and files... (and then namespaces, etc)

Re: Tokio and Prctl = Nasty Bug

#40

Post by fc417fc802 »

skissane wrote: post #28
fc417fc802 wrote: post #26

Earlier quoted context omitted.

More precisely, distinguishing a process and a thread is a pointless overspecification. Unfortunately POSIX mandates it and glibc accepts it. If you want to register per-thread signal handlers you're forced to step outside the bounds of glibc and pthreads which I think is quite unfortunate.

Digressing a little, but Glibc’s pthreads implementation is painful, because they don’t provide any public API to map a pthread_t to the kernel TID, except for the horrendously awful thread_db. Of course, for the current thread, you can just call gettid() - but if you want to map pthread_t to TID for another thread, the thread_db abomination is the only supported way. Bionic supplies a nice simple pthread_gettid_np()…

Huh I never really thought about that before. Seems like a glaring oversight but then again do any POSIX APIs even involve threads? Which itself illustrates the absurdity because what modern OS doesn't support multiple scheduling entities per virtual address space? Or should I have said per thread group? What was a process supposed to be again? (And what was the point of the thing?)

Digressing the conversation further, I notice in the docs that CLONE_SIGHAND requires CLONE_VM and CLONE_THREAD requires CLONE_SIGHAND. Any idea if there's a technical reason for that or is it just POSIX constraints needlessly infecting the kernel?

It's particularly confusing that the TID is the real identifier but the documentation generally refers to scheduling entities as processes. So you use a TID to refer to a process and a PID to refer to a thread group ... right. Very straightforward.

Post reply on HN