31–40 of 76 posts
The GNU coreutils timeout command encapsulates a lot of these issues. It's surprisingly difficult to handle all the edge cases and races. https://www.maizure.org/projects/decoded-gnu-coreutils/timeo...
This is why cgroups were invented. They solve this problem. Start a process in its own cgroup, and you can later confidently kill the process and all of its descendants. Container "technologies" use cgroups extensively, as does systemd service management.
Yes, for tracking processes and reliable resource control. Prior to cgroups, in Google's Borg cluster management daemon the best strategy I was able to come up with for reliably and efficiently tracking all the processes in a job was:
- assign each job a supplementary group id from a range reserved for Borg, and tag any processes that were forked into that job with that group id
- use a kernel netlink connector socket to follow PROC_EVENT_FORK events to find new processes/threads, and assign them to a job based on the parent process; if the parent process wasn't found for some reason then query the process' groups in /proc to find the Borg-added group id to determine which job it's a part of.
- if the state gets out of sync (due to a netlink queue overflow, or a daemon restart) do a full scan of /proc (generally avoided since the overhead for continually scanning /proc got really high on a busy machine).
That way we always have the full list of pids for a given group. To kill a job, nuke all the known processes and mark the group id as invalid, so any racy forks will cause the new processes to show up with a stale Borg group id, which will cause them to be killed immediately.
This approach might would have had trouble keeping up with a really energetic fork bomb, but fortunately Borg didn't generally have to deal with actively malicious jobs, just greedy/misconfigured ones.
Once we'd developed cgroups this got a lot simpler.
This is why cgroups were invented. They solve this problem. Start a process in its own cgroup, and you can later confidently kill the process and all of its descendants. Container "technologies" use cgroups extensively, as does systemd service management.
systemd uses cgroups, correct? just wondering what the options are for learning more about this, would it be enough, assuming you'd only be working with systemd operating systems, to learn the systemd concepts of slices etc.?
This is why cgroups were invented. They solve this problem. Start a process in its own cgroup, and you can later confidently kill the process and all of its descendants. Container "technologies" use cgroups extensively, as does systemd service management.
[CGroups original developer] Yes, for tracking processes and reliable resource control. Prior to cgroups, in Google's Borg cluster management daemon the best strategy I was able to come up with for reliably and efficiently tracking all the processes in a job was: - assign each job a supplementary group id from a range reserved for Borg, and tag any processes that were forked into that job with that group id - use a k…
Earlier quoted context omitted.
That's definitely the correct way to do this today. But even then `kill -9 $(< /sys/fs/cgroup/systemd/tasks)` is not enough if your goal is to reliably kill all processes because that's not atomic. Instead you'll have to freeze all processes, send SIGKILL and then unfreeze.
Can freezing be done atomically?
Really the best way to do it is to put it inside a PID namespaces and then kill the pid1. Unfortunately, most processes don't act correctly as a pid1 (the default signal mask is different for pid1, causing default "safe exit" signal behaviour to break for most programs). You could run a separate pid1 that just forwards signals (this is what Docker does with "docker run --init" and similar runtimes do the same thing). But now the solution has gotten significantly more complicated than "use PID namespaces".
Arguably the most trivial and workable solution is process groups and using a negative pid argument to kill(2), but that requires the processes to be compliant and not also require their own process groups. (I also haven't yet read TFA, it might say that this approach is also broken for reasons I'm not familiar with.)
Earlier quoted context omitted.
Can freezing be done atomically?
Not sure to be honest. From the documentation: "Writing "FROZEN" to the state file will freeze all tasks in the cgroup". Even if not, it should still be sufficient once all tasks are frozen: If you then send SIGKILL to all processes in the group, no fork bomb or similar process kerfuffle will be able to avoid being killed once they get unfrozen.
Thankfully Raymond Chen wrote an article[2] about Job Objects which helped me figure out the last bits. I genuinely am not sure I could have gotten it right without that article. There’s so many subtle ways to fail!
[1] https://github.com/bazelbuild/bazel-watcher/pull/144/files
[2] https://devblogs.microsoft.com/oldnewthing/20130405-00/?p=47...
Earlier quoted context omitted.
> Beyond the ability to unambiguously specify which process should be waited for, this change will eventually enable another interesting feature: it will make it possible to wait for a process that is not a child — something that waitid() cannot do now. Since a pidfd is a file descriptor, it can be passed to another process via an SCM_RIGHTS datagram in the usual manner. The recipient of a pidfd will, once this funct…
Perhaps pidfd_open(pid, ...)? https://lwn.net/Articles/789023/ I find it bizarre they called it "pidfd_" rather than just "process_" or "proc_"... almost seems like they deliberately avoided the obvious?
In fact it's exactly equivalent to getting a file descriptor for /proc/$pid -- Christian (the person who developed the patchsets) quite cleverly solidified a trick that some folks knew about for several years (that you could use /proc/$pid as a race-free way of checking if a process has died if you grabbed a handle while it was still alive). Before pidfd_send_signal(2) there wasn't a way to use that "interface" nicely. But now it's a first-class citizen (and Christian had to fight a lot of battles to get this in over several releases).
It's really cool work and I have high hopes it will be used far and wide because it solves so many individual problems in one fell swoop.
The GNU coreutils timeout command encapsulates a lot of these issues. It's surprisingly difficult to handle all the edge cases and races. https://www.maizure.org/projects/decoded-gnu-coreutils/timeo...
Cool. Anyone know how they make diagrams like those?