Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tokio and Prctl = Nasty Bug

kobzol.github.io

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Tokio and Prctl = Nasty Bug

#71

Post by fc417fc802 »

cryptonector wrote: post #69
fc417fc802 wrote: post #68

Earlier quoted context omitted.

Good point about multiprocessor systems. He did say this is being used on HPC clusters. You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. > have the `vfork()` happen in worker threads Fair enough. Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?

> Good point about multiprocessor systems. He did say this is being used on HPC clusters. Plus essentially all systems now are multiprocessor systems. > You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. You don't have to use much C here at all. > Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than c…

Note my nomenclature here is parent (doesn't block), worker (calls vfork), and child (calls execve).

If you used vfork then the worker doesn't unblock until execve has executed successfully (AFAIU). In that case if the exec succeeded then nothing will have been written whereas if it failed then the child will have written out the error code to the worker's stack before calling _exit.

The worker can then use an atomic write to communicate the ultimate result back to the parent in an asynchronous manner.

Re: Tokio and Prctl = Nasty Bug

#72

Post by ricardobeat »

AntiRush wrote: post #53
ricardobeat wrote: post #52

Earlier quoted context omitted.

There was no mention of a ten second interval anywhere in the code, only in the tests they wrote while debugging.

As mentioned in the article, there is a 10 second value in Tokio - the default thread timeout.

That wouldn’t be greppable in the source though?

Re: Tokio and Prctl = Nasty Bug

#73

Post by vlovich123 »

kobzol wrote: post #34
vlovich123 wrote: post #5

> Edit: Someone on Reddit sent me a link to a method that can override the thread keep-alive duration. Its description makes it clear why the tasks were failing after exactly 10 seconds > Yeah, testing if a task can run for 20 seconds isn’t great, but hey, at least it’s something Well a reasonable thing to me is then to use the override within the test to shorten it (e.g. to 1s & use a 2s timeout).

Could be done, yeah, but 20s isn't that much, and I'd like to avoid adding more test-only magic environment variables zo configure this (our end-to-end tests are in Python and they use HQ as a binary).

I appreciate the sentiment, but I think it’s making the wrong pragmatism/purity tradeoff. The test is brittle - what happens when a future update of the dependency in a couple of years makes a change to the default timeout value? Aside from making test runs quicker which is good for anyone running the test suite without caring about this 1 test itself, it future proofs the test flakiness better against defaults changing out from under you.

Re: Tokio and Prctl = Nasty Bug

#74

Post by cryptonector »

fc417fc802 wrote: post #71
cryptonector wrote: post #69

Earlier quoted context omitted.

> Good point about multiprocessor systems. He did say this is being used on HPC clusters. Plus essentially all systems now are multiprocessor systems. > You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. You don't have to use much C here at all. > Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than c…

Note my nomenclature here is parent (doesn't block), worker (calls vfork), and child (calls execve). If you used vfork then the worker doesn't unblock until execve has executed successfully (AFAIU). In that case if the exec succeeded then nothing will have been written whereas if it failed then the child will have written out the error code to the worker's stack before calling _exit. The worker can then use an atomic…

Ah, yes, that's true. I'm so used to using pipes like this that I forgot. You're quite right. Though I'd still use something like a pipe rather than a condition variable, say, because on Unix and Linux systems you can't have an event loop handle events on file descriptors _and_ condition variables.

Re: Tokio and Prctl = Nasty Bug

#75

Post by pengaru »

eqvinox wrote: post #63
pengaru wrote: post #62

Earlier quoted context omitted.

Can you actually substantiate your 85% confident claim? Because it doesn't ring the slightest bell here, and I don't see any mention of "deathsig" in glibc's LinuxThreads fork of Xavier's found @ https://ftp.gnu.org/gnu/libc/glibc-linuxthreads-2.5.tar.bz2 I used LinuxThreads back in the 90s, and its main problem ISTR was hijacking SIGUSR[12]. My interests back then involved demo programming using SVGAlib, and mixing…

> Can you actually substantiate your 85% confident claim? I unfortunately can't, it was apparently added in 2.1.57, which would be somewhere around 1998~1999. I started working with Linux around 2001~2002, and this association of PDEATHSIG with LinuxThreads has at some point embedded itself into my brain… I can't reconstruct when or why. And I can't seem to find the specific patch that added PDEATHSIG, and can't find…

Thanks for digging this up, that kerrisk pdf is great.

Re: Tokio and Prctl = Nasty Bug

#76

Post by bhawks »

kobzol wrote: post #32
bhawks wrote: post #13

Normally I'd stay away from job control posix APIs - but since HyperQueue is a job control system, it might be appropriate if the worker was a session leader. If it dies than all its subprocesses would receive SIGHUP - which is fatal by default. Generally you'd use this functionality to implement something like sshd or an interactive shell. HQ seems roughly analogous. https://notes.shichao.io/apue/ch9/#sessions

I do use setsid when spawning the children (I omitted it from the post, but I set it in the dsme pre_exec call where I configure DEATHSIG) but they don't receive any signal, IIRC. Or if they do, it does not seem to be propagated to their children.

Calling setsid in pre_exec isn't exactly correct, you'd do this to daemonize a process in order to prevent it from getting signals from the process which spawned you exited or its terminal disconnected.

If you read exit(3) you'll see that you also need a controlling terminal for the kernel to send SIGHUP to the processes in your foreground process group.

In python it'd look roughly like:

  master, slave = os.openpty()

  # ensure we can call setsid successfully

  try:

    pid = os.fork()

    if pid > 0:

      os.close(master)

      os.close(slave)

      sys.exit(0)

  finally:

    sys.exit(1)

  os.setsid()
  # we are now session leader and process group leader

  fcntl.ioctl(master, termios.TIOCSCTTY, 0)
  # we now have a controlling pty - closing master will send us a SIGHUP

  os.close(slave)

  # go start spawning subprocesses - if _this_ process is killed, they will receive SIGHUPs. Unless they take themselves out of your session or the foreground process group.

Re: Tokio and Prctl = Nasty Bug

#77

Post by cryptonector »

cryptonector wrote: post #74
fc417fc802 wrote: post #71

Earlier quoted context omitted.

Note my nomenclature here is parent (doesn't block), worker (calls vfork), and child (calls execve). If you used vfork then the worker doesn't unblock until execve has executed successfully (AFAIU). In that case if the exec succeeded then nothing will have been written whereas if it failed then the child will have written out the error code to the worker's stack before calling _exit. The worker can then use an atomic…

Ah, yes, that's true. I'm so used to using pipes like this that I forgot. You're quite right. Though I'd still use something like a pipe rather than a condition variable, say, because on Unix and Linux systems you can't have an event loop handle events on file descriptors _and_ condition variables.

And those are my favorite signalling mechanisms: async I/O and condition variables.

Re: Tokio and Prctl = Nasty Bug

#78

Post by fc417fc802 »

cryptonector wrote: post #74
fc417fc802 wrote: post #71

Earlier quoted context omitted.

Note my nomenclature here is parent (doesn't block), worker (calls vfork), and child (calls execve). If you used vfork then the worker doesn't unblock until execve has executed successfully (AFAIU). In that case if the exec succeeded then nothing will have been written whereas if it failed then the child will have written out the error code to the worker's stack before calling _exit. The worker can then use an atomic…

Ah, yes, that's true. I'm so used to using pipes like this that I forgot. You're quite right. Though I'd still use something like a pipe rather than a condition variable, say, because on Unix and Linux systems you can't have an event loop handle events on file descriptors _and_ condition variables.

I thought the optimization goal was latency on the parent thread so I had in mind a bare atomic_store_explicit with memory_order_relaxed. However you wouldn't care about latency in the first place if you weren't scaling this up and obviously scaling up requires additional infrastructure that I'm ignoring here.

That said the linked article is using spawn_blocking from tokio meaning you can just call vfork directly and return the result. No need to roll your own in this case.

> you can't have an event loop handle events on file descriptors _and_ condition variables

Since events crossing the kernel boundary aren't fast to begin with I prefer to use a secondary event loop for the sluggish stuff and feed the output to the lower latency one.

Post reply on HN