Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tokio and Prctl = Nasty Bug

kobzol.github.io

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Tokio and Prctl = Nasty Bug

#61

Post by cryptonector »

fc417fc802 wrote: post #29
cryptonector wrote: post #23

If you want to be able to spawn processes that fask then `fork()` is NOT your friend. You want either `vfork()` (or `clone()` equivalent) or `posix_spawn()`. `fork()` is inherently very slow due to the need to either copy the VM of the parent, or arrange to copy pages on write, or copy the resident set of the parent (then copy any pages paged-in when those page-in events happen -- all three of these options are very…

Copying the page table isn't free but it isn't particularly expensive either. At least unless the parent is a real behemoth. Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.) The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so…

CoW is extremely expensive for threaded processes on multi-processor systems since you need TLB shootdowns.

> The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose?

Yes, so have the `vfork()` happen in worker threads.

> Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.)

I have proof that `vfork()` can be used safely: the several `posix_spawn()` implementations that use it.

Re: Tokio and Prctl = Nasty Bug

#62

Post by pengaru »

eqvinox wrote: post #12

> It is called PR_SET_DEATHSIG, and we configure it when spawning tasks using the prctl syscall like this PDEATHSIG was to my knowledge (85% confidence) created for the original Linux userspace pthreads implementation (LinuxThreads¹, before NPTL) that was created back when it was implemented via kernel processes (the kernel had no concept of threads yet). This is AFAIK also why it behaves oddly in regards to later-ad…

Can you actually substantiate your 85% confident claim? Because it doesn't ring the slightest bell here, and I don't see any mention of "deathsig" in glibc's LinuxThreads fork of Xavier's found @ https://ftp.gnu.org/gnu/libc/glibc-linuxthreads-2.5.tar.bz2

I used LinuxThreads back in the 90s, and its main problem ISTR was hijacking SIGUSR[12]. My interests back then involved demo programming using SVGAlib, and mixing LinuxThreads with SVGAlib was a mess due to both wanting to use SIGUSR1. Endless corrupt consoles...

Re: Tokio and Prctl = Nasty Bug

#63

Post by eqvinox »

pengaru wrote: post #62
eqvinox wrote: post #12

> It is called PR_SET_DEATHSIG, and we configure it when spawning tasks using the prctl syscall like this PDEATHSIG was to my knowledge (85% confidence) created for the original Linux userspace pthreads implementation (LinuxThreads¹, before NPTL) that was created back when it was implemented via kernel processes (the kernel had no concept of threads yet). This is AFAIK also why it behaves oddly in regards to later-ad…

Can you actually substantiate your 85% confident claim? Because it doesn't ring the slightest bell here, and I don't see any mention of "deathsig" in glibc's LinuxThreads fork of Xavier's found @ https://ftp.gnu.org/gnu/libc/glibc-linuxthreads-2.5.tar.bz2 I used LinuxThreads back in the 90s, and its main problem ISTR was hijacking SIGUSR[12]. My interests back then involved demo programming using SVGAlib, and mixing…

> Can you actually substantiate your 85% confident claim?

I unfortunately can't, it was apparently added in 2.1.57, which would be somewhere around 1998~1999. I started working with Linux around 2001~2002, and this association of PDEATHSIG with LinuxThreads has at some point embedded itself into my brain… I can't reconstruct when or why. And I can't seem to find the specific patch that added PDEATHSIG, and can't find a versioned history of LinuxThreads either…

Probably best to treat my comment as "grandpa tells weird stories that may or may not be true" :'(

[… I'm not even that old T_T]

Best reference I can find is in MAINTAINERS:

  N: Richard E. Gooch
  E: rgooch@atnf.csiro.au
  D: parent process death signal to children
  D: prctl() syscall
  S: CSIRO Australia Telescope National Facility
  S: P.O. Box 76, Epping
  S: N.S.W., 2121
  S: Australia
[ed.] wait! — https://man7.org/conf/piter2019/once_upon_an_API-Linux-Piter...

"(Of course, there was no explanation of why the feature was needed)"

Note Richard was at minimum involved in discussions about LinuxThreads: https://lkml.iu.edu/hypermail/linux/kernel/9806.2/1227.html — and it does mention prctl and "dying main thread"…

Re: Tokio and Prctl = Nasty Bug

#64

Post by zamalek »

kobzol wrote: post #35
deathanatos wrote: post #22

> In particular, it is not always possible for HQ to ensure that when a process that spawns tasks (called worker) quits unexpectedly (e.g. when it receives SIGKILL), its spawned tasks will be cleaned up. Sadly, Linux does not seem to provide any way of implementing perfect structured process management in user space. In other words, when a parent process dies, it is possible for its (grand)children to continue execut…

Yeah by user space I just meant without root, sorry. HQ runs on supercomputers where the environment is heavily locked up, even Docker doesn't work. I think that PID namespaces aren't really possible, but I haven't tried it yet. Subreaper doesn't help, because if the worker dies, the children aren't killed, even if they are the children of the worker, they will be just reparented to init.

FWIW you can unshare PID and user at the same time: https://github.com/porkg/porkg/blob/rs/crates/porkg-linux/sr...

If you don't care about being able to use different uids and gids then simply become root in the new namespace: https://github.com/porkg/porkg/blob/rs/crates/porkg-linux/sr... . Root inside the namespace will then be equivalent to the original uid+gid outside.

I am using clone, which has the very important caveat: more than one thread running is UB. That's why I use a zygote (a process forked from the root very early on - i.e. before starting the tokio runtime). You can probably avoid all of that by using exec+unshare.

But, given you're running on old kernels and constrained environments this may be not possible at all. Maybe make it configurable?

Re: Tokio and Prctl = Nasty Bug

#65

Post by eqvinox »

fulafel wrote: post #24
eqvinox wrote: post #12

> It is called PR_SET_DEATHSIG, and we configure it when spawning tasks using the prctl syscall like this PDEATHSIG was to my knowledge (85% confidence) created for the original Linux userspace pthreads implementation (LinuxThreads¹, before NPTL) that was created back when it was implemented via kernel processes (the kernel had no concept of threads yet). This is AFAIK also why it behaves oddly in regards to later-ad…

There's been no fundamental change in the kernel level representation of pthreads, they are still clone()d processes with just some sharing flags set differently that eg affect how PIDs work.

> they are still clone()d processes with just some sharing flags set differently that eg affect how PIDs work.

I'd say this is depending on perspective both true and false¹, but also unhelpful to work with here.

Instead, I would suggest this perspective: the kernel has neither processes nor threads; it has tasks, which are entities the scheduler can run. They're exposed to userland as processes and threads. Excluding kernel tasks/threads, which can have arbitrary rules but are also user-visible, a task is exposed as a thread, and a set of threads is exposed as a process. Both operations working with threads as well as operations working with processes exist.

We're looking at an API in this case that works with threads on one side (parent, the signal is triggered by thread exit) and processes on the other (child, the signal is process-targeted). How these were created is irrelevant, what matters is the abstractions they refer to.

¹ you could equally well argue that processes do not exist in the kernel, they're just threads with sharing flags set differently.

Re: Tokio and Prctl = Nasty Bug

#66

Post by LegionMammal978 »

jerf wrote: post #57

A similar issue in Go, that I've encountered in real code: https://github.com/golang/go/issues/27505#issuecomment-71370... In a nutshell, if you want to use the death signal, which is very handy and useful, you also need to lock an OS thread so that it can't be destroyed. Fortunately I'm only spawning one process so I don't need to jump through hoops, I can just dedicate a thread to it, but it would be inconvenient t…

Yeah, a lot of process vs. thread distinctions can be unclear, even in documentation. E.g., the Linux clone(2) man page mostly talks about "the child process", even though it can create either a new thread or a new process.

The weirdest case of processes vs. threads is definitely the setuid() family of functions on Linux. The underlying syscalls apply the new uid (or euid, fsuid, etc.) to the current thread, but POSIX requires them to apply to the entire process. How does glibc paper over this? It registers a realtime signal handler which calls the appropriate syscall, and sends that signal to every thread when the wrapper function is called. On top of that, it quietly removes the handler's signal number (SIGSETXID) from all calls to sigfillset(), sigprocmask(), and pthread_sigmask() to keep it from getting blocked, and bumps the value of SIGRTMIN in the userspace headers so that programs won't notice the gap. I believe musl libc does something very similar.

Re: Tokio and Prctl = Nasty Bug

#67

Post by LegionMammal978 »

zamalek wrote: post #64
kobzol wrote: post #35

Earlier quoted context omitted.

Yeah by user space I just meant without root, sorry. HQ runs on supercomputers where the environment is heavily locked up, even Docker doesn't work. I think that PID namespaces aren't really possible, but I haven't tried it yet. Subreaper doesn't help, because if the worker dies, the children aren't killed, even if they are the children of the worker, they will be just reparented to init.

FWIW you can unshare PID and user at the same time: https://github.com/porkg/porkg/blob/rs/crates/porkg-linux/sr... If you don't care about being able to use different uids and gids then simply become root in the new namespace: https://github.com/porkg/porkg/blob/rs/crates/porkg-linux/sr... . Root inside the namespace will then be equivalent to the original uid+gid outside. I am using clone, which has the very import…

Ubuntu [0] and some other distros have been trending towards disabling unprivileged user namespaces, unless you have specific AppArmor capabilities or other such mechanisms. So it's not something you can count on being available, unfortunately. (At least, not without jumping through many hoops to satisfy every distro's maintainers.) I've also had some ideas that have been stymied by a lack of unprivileged user namespaces.

[0] https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged...

Re: Tokio and Prctl = Nasty Bug

#68

Post by fc417fc802 »

cryptonector wrote: post #61
fc417fc802 wrote: post #29

Earlier quoted context omitted.

Copying the page table isn't free but it isn't particularly expensive either. At least unless the parent is a real behemoth. Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.) The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so…

CoW is extremely expensive for threaded processes on multi-processor systems since you need TLB shootdowns. > The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose? Yes, so have the `vfork()` happen in worker threads. > Unless you enjoy footguns posix_spawn is probab…

Good point about multiprocessor systems. He did say this is being used on HPC clusters.

You can also use C safely if you're careful. Doesn't mean it isn't full of footguns.

> have the `vfork()` happen in worker threads

Fair enough. Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?

Re: Tokio and Prctl = Nasty Bug

#69

Post by cryptonector »

fc417fc802 wrote: post #68
cryptonector wrote: post #61

Earlier quoted context omitted.

CoW is extremely expensive for threaded processes on multi-processor systems since you need TLB shootdowns. > The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose? Yes, so have the `vfork()` happen in worker threads. > Unless you enjoy footguns posix_spawn is probab…

Good point about multiprocessor systems. He did say this is being used on HPC clusters. You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. > have the `vfork()` happen in worker threads Fair enough. Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?

> Good point about multiprocessor systems. He did say this is being used on HPC clusters.

Plus essentially all systems now are multiprocessor systems.

> You can also use C safely if you're careful. Doesn't mean it isn't full of footguns.

You don't have to use much C here at all.

> Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?

The point is to signal that `execve(...)` did not fail but did start the new program -- but, how? You can use a condition variable or whatever to signal failure to exec, but you can't run code after a successful exec that would signal a condition variable or whatever because the program that was calling exec is now not running. A close-on-exec pipe serves to asynchronously signal success to the parent: it only closes either when the exec succeeds or after the exec fails and you close it after signaling the failure however you like. This works because the "code that runs after the exec succeeds" here is not code in the new program but code in the kernel after the kernel commits to not returning from `execve()`!

What else has this property? Only file descriptors, or you can insist that the program you exec must signal some how that it started. But the latter is intrusive, while the former is not. If it's file descriptors then it has to be something that supports async I/O, and the simplest thing would be a pipe or a socketpair. Now if you're going to use a pipe for exec success reporting then you might as well also use it to signal failure because why have two mechanisms, one for reporting success and one for reporting failure?

You still have to deal with the process' eventual exit, and reporting that asynchronously along with the exit status. But you get to report on all of three distinct events:

  - exec failure (e.g., ENOENT)
  - exec start (really *exec didn't fail*)
  - child process exit/death
You could choose to not report exec start, I suppose. But you often want to know that exec didn't fail, and the only ways to know that are to either wait for the process to exit (which could be a long time!), assume that if enough time has passed then the exec did not fail (a lame heuristic), or just arrange to signal exec success. Since it is possible and easy to signal exec success via a close-on-exec pipe/socketpair then you might as well just do that.
Post reply on HN