Runit – a Unix init scheme with service supervision • Hacker News Board

Re: Runit – a Unix init scheme with service supervision

#31

Post by vezzy-fnord » Fri, Aug 14, 2015, 11:58 PM UTC

dap wrote: post #30

ori_b wrote: post #29

Earlier quoted context omitted.

What if init doesn't exit, and just hangs? What if it just goes crazy and starts erronously restarting your processes? There are more failure modes than simply crashing. At some point, you just have to assume that some critical components are working correctly. Adding complexity just makes it harder to reason about it, or, depending on how paranoid you are, prove it.

That's a slippery slope argument: because we can't solve the halting problem or verify program correctness, we shouldn't try to handle crashes, either? Agreed on minimizing complexity. The only part of the chain I described that's very complex is svc.startd, and that's largely to support rich configuration. Also don't mistake my position for saying that quality isn't important. Rather, just that perfection is not a r…

At least on Linux, PID1's death is an instant kernel panic. As such, it's wise to keep any service management logic out of it.

If the svscan process dies, then your system is still chugging along and you can intervene to restore the supervision tree (otherwise svscan inspects supervise processes at a regular 5s interval). If you have some really critical process, then you could integrate a checkpointer into the run script chain so that you can just pick off from the last image of the process state with minimal interruption.

Re: Runit – a Unix init scheme with service supervision

#32

Post by freshhawk » Sat, Aug 15, 2015, 12:19 AM UTC

paulsmith wrote: post #4

atom_enger wrote: post #2

Runit is amazing. I've used it on several large scale websites with great success. Runit follows the unix philosophy of being stupid simple and doing one thing incredibly well. If you're starting up a new project, consider using Runit.

Not only is it simple and does the do-the-one-thing-well thing well, which is true, it's also _correct_. It's hard to overstate how valuable runit is in production because of that. It does the right thing with regard to clearing the environment, detaching from controlling terminal, logging, and many other subtle aspects of operating a service. I never worry about runit.

Back when I changed from supervisord to runit my life was substantially improved. That correctness means way fewer emergency maintenance ops issues in production.

Re: Runit – a Unix init scheme with service supervision

#33

Post by ori_b » Sat, Aug 15, 2015, 12:52 AM UTC

dap wrote: post #30

ori_b wrote: post #29

Earlier quoted context omitted.

What if init doesn't exit, and just hangs? What if it just goes crazy and starts erronously restarting your processes? There are more failure modes than simply crashing. At some point, you just have to assume that some critical components are working correctly. Adding complexity just makes it harder to reason about it, or, depending on how paranoid you are, prove it.

That's a slippery slope argument: because we can't solve the halting problem or verify program correctness, we shouldn't try to handle crashes, either? Agreed on minimizing complexity. The only part of the chain I described that's very complex is svc.startd, and that's largely to support rich configuration. Also don't mistake my position for saying that quality isn't important. Rather, just that perfection is not a r…

> Also don't mistake my position for saying that quality isn't important. Rather, just that perfection is not a reasonable constraint.

At some point, for some component or set of components, perfection is your only choice, regardless of the rest of your design. At least when you consider a single node with a single point of failure; this is less true for a distributed system where you have redundancy.

At some point, you have to assume that either init is perfect, or that the code in the kernel to detect init failures is perfect, or that the watchdog monitoring the kernel is perfect, or whatever other layering you choose to put in place is perfect.

In a system with a finite number of components, there is always going to be a point at which you just say "this bit is going to have to be correct, and there's no other way around it".

Re: Runit – a Unix init scheme with service supervision

#34

Post by atom_enger » Sat, Aug 15, 2015, 12:58 AM UTC

manas952 wrote: post #16

Runit is fantastic. If you are using Chef - the runit cookbook integrates very nicely. https://supermarket.chef.io/cookbooks/runit

Now that I think of it, the chef cookbook for Runit is what made it so easy to deploy to production. The 'runit_service' resource was absolutely invaluable. Forget the complicated upstart stanzas or dealing with supervisord, just write a shell script to run your program in the proper environment and ba! You've got a service!

Re: Runit – a Unix init scheme with service supervision

#35

Post by akira2501 » Sat, Aug 15, 2015, 1:08 AM UTC

dap wrote: post #28

ori_b wrote: post #26

Earlier quoted context omitted.

Make it simple and obviously correct, and don't crash.

Just don't make any mistakes? Was that a joke? There are many reasons a process can die that are outside of its control, including signals from outside the process, handled (but uncorrectable) memory errors, and the OOM killer (on Linux). Besides that, it seems like a major design shortcoming if fatal errors in any particular program (however critical and however simple that program may be) can be unrecoverable for t…

If you're having hardware memory issues, your system is already in an undefined and unstable state.

If you send it a SIGTERM, it runs your shutdown scripts and reboots. If you send it a SIGKILL, your kernel will panic. As far as I remember, this isn't any different from init.

The OOM killer will _never_ take PID 1.

In runit all PID 1 has to do is run the service scanner. All that has to do is open directories and fork/exec the individual service managers. If it fails, it will try again in 1 second, forever. No complex logic needed. Just keep trying. In practice, it works surprisingly well.

If the individual service managers fail to run the startup script, it will try again in 1 second, forever. It works very will for most situations, but you _can_ customize this behavior easily. This simplicity is really helpful in an actual emergency because you don't get emergent behavior, like init deciding that your service is flapping and holding it down for 5 minutes.

Anecdotally, I've been using runit exclusively on all my systems (around 25 physical systems and 20-100 virtual ones depending) for at least 8 years now and I've never had a single issue.

The biggest problem I have with the design is that it puts your log services in a second level "behind" your main services, so you can sometimes miss that your log service failed to startup for some reason. This can be a real pain if your service uses blocking IO for it's stdout/stderr logging as it can cause the service to hang seemingly without explanation.

Re: Runit – a Unix init scheme with service supervision

#36

Post by dap » Sat, Aug 15, 2015, 1:21 AM UTC

ori_b wrote: post #33

dap wrote: post #30

Earlier quoted context omitted.

That's a slippery slope argument: because we can't solve the halting problem or verify program correctness, we shouldn't try to handle crashes, either? Agreed on minimizing complexity. The only part of the chain I described that's very complex is svc.startd, and that's largely to support rich configuration. Also don't mistake my position for saying that quality isn't important. Rather, just that perfection is not a r…

> Also don't mistake my position for saying that quality isn't important. Rather, just that perfection is not a reasonable constraint. At some point, for some component or set of components, perfection is your only choice, regardless of the rest of your design. At least when you consider a single node with a single point of failure; this is less true for a distributed system where you have redundancy. At some point,…

I think you've misunderstood my point. The design I described does not require any component to be perfect. If any of these components (including the kernel) crashes, the system _still_ converges to the correct state.

Re: Runit – a Unix init scheme with service supervision

#37

Post by dkubb » Sat, Aug 15, 2015, 1:29 AM UTC

edwintorok wrote: post #19

There is a good comparison/documentation of what an init system like runit should do and why on the S6 site: http://skarnet.org/software/s6/why.html http://skarnet.org/software/s6/overview.html Runit has the advantage that it is packaged in Debian and you can start using it right away.

Do you have any experience with S6? Do you know how it compares to runit?

Re: Runit – a Unix init scheme with service supervision

#38

Post by ori_b » Sat, Aug 15, 2015, 1:30 AM UTC

dap wrote: post #36

ori_b wrote: post #33

Earlier quoted context omitted.

> Also don't mistake my position for saying that quality isn't important. Rather, just that perfection is not a reasonable constraint. At some point, for some component or set of components, perfection is your only choice, regardless of the rest of your design. At least when you consider a single node with a single point of failure; this is less true for a distributed system where you have redundancy. At some point,…

I think you've misunderstood my point. The design I described does not require any component to be perfect. If any of these components (including the kernel) crashes, the system _still_ converges to the correct state.

You missed my point. At the moment, you're assuming that the kernel behavior will reliably, buglessly fall into one of two outcomes: Either lossless full system reset, or detect init has failed and restart it. You're ignoring the possibility of, deadlocks, failures in detecting that init has crashed, bugs in the special casing of init to restart it, etc. You are assuming that there are components that do certain things perfectly reliably.

You haven't gotten rid of a correctness assumption, you've just shuffled it around a bit.

Re: Runit – a Unix init scheme with service supervision

#39

Post by keypusher » Sat, Aug 15, 2015, 2:29 AM UTC

akira2501 wrote: post #35

dap wrote: post #28

Earlier quoted context omitted.

Just don't make any mistakes? Was that a joke? There are many reasons a process can die that are outside of its control, including signals from outside the process, handled (but uncorrectable) memory errors, and the OOM killer (on Linux). Besides that, it seems like a major design shortcoming if fatal errors in any particular program (however critical and however simple that program may be) can be unrecoverable for t…

If you're having hardware memory issues, your system is already in an undefined and unstable state. If you send it a SIGTERM, it runs your shutdown scripts and reboots. If you send it a SIGKILL, your kernel will panic. As far as I remember, this isn't any different from init. The OOM killer will _never_ take PID 1. In runit all PID 1 has to do is run the service scanner. All that has to do is open directories and for…

This is one of the reasons it's common to run a watchdog in HA critical systems. If something fails in supervision or at a low level and nothing is responding, nobody is around to tickle the watchdog and the entire system reboots.

Re: Runit – a Unix init scheme with service supervision

#40

Post by eddieroger » Sat, Aug 15, 2015, 4:10 AM UTC

manas952 wrote: post #16

Runit is fantastic. If you are using Chef - the runit cookbook integrates very nicely. https://supermarket.chef.io/cookbooks/runit

This was how I got introduced to runit, and I couldn't agree more. It's really amazing at what it does, and quickly has become my favorite way of keeping processes running. It's just so easy.