Live data from Hacker News

Runit – a Unix init scheme with service supervision

smarden.org

51–60 of 69 posts

Re: Runit – a Unix init scheme with service supervision

#51
post #43

I use GNU dmd instead. Simple and very extensible with Scheme. I use it as PID 1 on 2 of my machines, but I use another instance of it as a user service manager on all of my machines. I've also been meaning to replace runit with dmd in Phusion's passenger-docker image to get something more hackable. https://gnu.org/s/dmd

Having dynamic memory allocation, let alone garbage collection, in my pid 1 doesn't sounds like a great idea. (I know, as long as I'm using Linux, I'm stuck with dynamic memory allocation in the fucking kernel . I don't have a plan for how to fix that yet.) (Also, thank you so much for all your help with Guix this last week!)

There's nothing wrong with dynamic allocation in a kernel. It is for example better than having fixed size process tables and all the crap that comes with that.

"holy crap I've got to recompile my kernel to get more processes" is so 1995...

Re: Runit – a Unix init scheme with service supervision

#53
post #30
post #29

Earlier quoted context omitted.

What if init doesn't exit, and just hangs? What if it just goes crazy and starts erronously restarting your processes? There are more failure modes than simply crashing. At some point, you just have to assume that some critical components are working correctly. Adding complexity just makes it harder to reason about it, or, depending on how paranoid you are, prove it.

That's a slippery slope argument: because we can't solve the halting problem or verify program correctness, we shouldn't try to handle crashes, either? Agreed on minimizing complexity. The only part of the chain I described that's very complex is svc.startd, and that's largely to support rich configuration. Also don't mistake my position for saying that quality isn't important. Rather, just that perfection is not a r…

> we can't solve the halting problem or verify program correctness

We can't solve halting problem or verify program correctness for all programs. For a large subset of all programs, you are able to do both. This is a very important distinction.

We even have verified C compilers (CompCert). Writing a verified service manager should be easier in comparison.

Re: Runit – a Unix init scheme with service supervision

#54
post #44

Earlier quoted context omitted.

I appreciate the detailed reply, and the others as well. To address your questions and some of the others: We can generally pin our entire fleet against whatever we want -- until we can't. One particular (very large, very important) dependency currently requires CentOS 6.5. Go figure. As much as possible, I want to treat infrastructure as a commodity, so catering to specific versions (even LTR versions) breaks that g…

> One particular (very large, very important) dependency currently requires CentOS 6.5. What dependency is that? And can you run it in a single-purpose virtual machine, rather than directly on real hardware, to make it easier to manage? > As much as possible, I want to treat infrastructure as a commodity, so catering to specific versions (even LTR versions) breaks that goal. As another reply notes, if we lock in at U…

I'm wary of naming the dependency, because it's a vendor's solution and I'm not trying to call them out, but it's their ecosystem of tools and it has to run as the host OS, not virtualized. But all your comments are spot-on, and I appreciate it.

The Upstart thing is just somewhat odd, because it's a Canonical project, and Ubuntu is also Canonical's. At any rate, you're right that it's not immediately lost. But trying to pin against even the long-term releases of Ubuntu seems hopeless. I've been working on upgrading a 12.04 deployment to 14.04. There aren't any features that we're missing in the OS, so it's really only a concern from a security or support perspective. The shop I'm in was already using Ubuntu when I came on, but maybe the lesson here is to migrate to Debian, rather than bother upgrading to the newer Ubuntu release. (Migrating to CentOS would be somewhat more work.)

Re: Runit – a Unix init scheme with service supervision

#55
A genuine question, as it's not quite clear to me: can someone explain like I'm 5, what can I use runit for exactly? Is it a replacement for init.d or something more like monit? I always use monit to keep my Rails-related processes running and I've read somewhere that using monit combined with runit may be a better solution, but isn't it a bit redundant to have the two at the same time?

Re: Runit – a Unix init scheme with service supervision

#56
post #54

Earlier quoted context omitted.

> One particular (very large, very important) dependency currently requires CentOS 6.5. What dependency is that? And can you run it in a single-purpose virtual machine, rather than directly on real hardware, to make it easier to manage? > As much as possible, I want to treat infrastructure as a commodity, so catering to specific versions (even LTR versions) breaks that goal. As another reply notes, if we lock in at U…

I'm wary of naming the dependency, because it's a vendor's solution and I'm not trying to call them out, but it's their ecosystem of tools and it has to run as the host OS, not virtualized. But all your comments are spot-on, and I appreciate it. The Upstart thing is just somewhat odd, because it's a Canonical project, and Ubuntu is also Canonical's. At any rate, you're right that it's not immediately lost. But trying…

> The Upstart thing is just somewhat odd, because it's a Canonical project, and Ubuntu is also Canonical's.

That's why they pushed so hard for Debian to use Upstart. Once Debian switched to systemd instead, Canonical announced that they would too.

> The shop I'm in was already using Ubuntu when I came on, but maybe the lesson here is to migrate to Debian, rather than bother upgrading to the newer Ubuntu release.

I'd certainly recommend that myself. Unless you have a hard requirement for an "Enterprise" distribution (RHEL or SLES), I tend to advocate Debian stable rather than Ubuntu LTS, especially on a server.

Re: Runit – a Unix init scheme with service supervision

#57
post #37

Earlier quoted context omitted.

Do you have any experience with S6? Do you know how it compares to runit?

I've used s6 a lot, though not as an init replacement in any systems that matter. As a supervisor I think it's great. For basic stuff (root process supervising supervisors supervising daemons) the two are basically identical with some superficial differences. For wider ranging stuff s6 is really good since it has a bunch of ancillary programs that solve a lot of serious issues in full system supervision (readyness no…

[deleted]

Re: Runit – a Unix init scheme with service supervision

#58
post #37

Earlier quoted context omitted.

Do you have any experience with S6? Do you know how it compares to runit?

I've used s6 a lot, though not as an init replacement in any systems that matter. As a supervisor I think it's great. For basic stuff (root process supervising supervisors supervising daemons) the two are basically identical with some superficial differences. For wider ranging stuff s6 is really good since it has a bunch of ancillary programs that solve a lot of serious issues in full system supervision (readyness no…

I missed actually answering the question about the difference between the two (plus, that comment is in dire need of editing and the edit window appears to have expired).

The superficial differences are things like: service to logger pipe holding happens at different levels of the supervision tree (the root `s6-svscan' holds them in s6, the per-service runsv holds it in runit), `s6-svc -CMD behavior cannot currently be overridden whereas you can with `sv CMD', `s6-svscan' will immediately re-scan its directory with SIGALRM whereas `runsvdir' only polls for changes on a 5 second timer.

For basic supervision tasksboth are great, with runit being the simpler of the two in terms of understanding what it gives out of the box. For larger tasks (full system supervision, inter-service ordering dependencies, etc) s6 has the tools to make that easy whereas with runit you're going to find yourself playing stupid tricks in run scripts to get similar behavior.

Re: Runit – a Unix init scheme with service supervision

#59
post #55

A genuine question, as it's not quite clear to me: can someone explain like I'm 5, what can I use runit for exactly? Is it a replacement for init.d or something more like monit? I always use monit to keep my Rails-related processes running and I've read somewhere that using monit combined with runit may be a better solution, but isn't it a bit redundant to have the two at the same time?

runit will start a daemon, and if it exits, restart it. i.e. you have runit sat 'atop' your daemon, as it were.

monit checks for things, one of which is 'is the daemon running', and can then do things about it.

runit to keep things that crash occasionally running and monit to keep an eye on general system health would seem totally sensical to me.

(see also s6 which is another runit-like)

Re: Runit – a Unix init scheme with service supervision

#60
post #55

A genuine question, as it's not quite clear to me: can someone explain like I'm 5, what can I use runit for exactly? Is it a replacement for init.d or something more like monit? I always use monit to keep my Rails-related processes running and I've read somewhere that using monit combined with runit may be a better solution, but isn't it a bit redundant to have the two at the same time?

Essentially it's an alternative method for starting long running services (and restarting them if they fail).

The things all daemontools-inspired process supervisors do: Runs a scanner gainst a service directory containing a directory for each program you want supervised (classically these are symlinks to another part of the system). The scanner spawns a supervisor program for every directory it finds which then looks for a program called `run' inside that directory and runs it as a foreground child. If the supervisor's child stops, it runs an optional script in it's supervision directory called `finish', and then runs run again. If one of the scanner's child supervisors stops, the scanner spawns it again.

In runit's case, the scanner is called `runsvdir', the supervisors are called `runsv', and it also comes with a program called `runit' that can act as a replacement for your PID1 init whose sole job outside of boot and shutdown time is to resurrect runsvdir if it exits. Note that runsvdir is perfectly happy to be run via an entry in your /etc/inittab if you're running under sysvinit.

At their most simplistic, that's all process supervision is - a bomb-proof way of keeping services running. All supervisors in the daemontools family come with a control program to interact with the process supervisor, as well as a stdin-based logger that doesn't rely on syslog. The main benefits it brings over the classic init.d/ model are simplicity (the most complex run script I have is 14 lines of dead simple shell, most are 4-5) and automatic restartability which init.d/ daemonization doesn't have.

From what I understand from monit's manpage, it's a full-bore rule-basd system monitor. Generally speaking, process supervision is tacking one problem (keep daemons running), whereas monit is tacking another (system state monitoring). Yes monit can act as a process supervisor, but it does so by polling the system state and hooking into the existing daemonization infrastructure.

Post reply on HN