Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing systemd units that stop gracefully before shutdown

psdn.io

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Writing systemd units that stop gracefully before shutdown

#41

Post by tokenrove »

empthought wrote: post #18
bravetraveler wrote: post #14

Earlier quoted context omitted.

While I agree it's a bit of a weird take, for example -- there may be performance tradeoffs made in any given workload to make the disk consistent, inconsistently The 'most' there is doing some effort It is actually quite a common practice for those being audited for disaster recovery to do exactly that -- yank cables. More realistically, flip some switches We do it once a year, set aside a region and time... then te…

Right, there are failure modes that have to be tested and accounted for, and one of them is the state being inconsistent after a shutdown. The previous poster seemed to advocate for not thinking of this as a failure mode at all but rather normal operation, which I just don’t see as true.

This paper was influential with regards this idea: https://www.usenix.org/conference/hotos-ix/crash-only-softwa...

I don't think it's that unusual, but obviously there are tradeoffs.

Re: Writing systemd units that stop gracefully before shutdown

#42

Post by kentonv »

empthought wrote: post #13
SoftTalker wrote: post #11

I fundamentally disagree with the idea that software should require or even expect a graceful shutdown. You can never stop the user from yanking the power cord out of the socket, which is what they will do if you force a bunch of housekeeping to happen before shutdown. You have to deal with crash/power failure recovery anyway. So do your housekeeping on startup. Shutdown should be a quick and simple termination.

This is a weird take; most systems in data centers don’t have people walking from rack to rack yanking power cords, and most consumer systems don’t even have a power cord to yank.

Weird take?

It is a table stakes expectation for most servers that they will not lose data when the power goes out, or when the kernel panics, or when the server itself crashes or runs out of memory. If your software requires graceful shutdown, that seems to imply that it will lose data in all those cases.

You can perhaps use graceful shutdown to perform some optimization that allows subsequent startup to go faster, e.g. put things in a clean state that avoids the need for a recovery pass on the next startup... but these days with good journaling techniques "recovery" is generally very fast. When that's the case, it's arguably better to always perform non-graceful shutdown to make sure you are actually testing your recovery code, otherwise it might turn out not to work when you need it.

So yeah, I agree with SoftTalker. Assume all shutdowns will be sudden and unexpected, and design your code to cope with that.

Re: Writing systemd units that stop gracefully before shutdown

#43

Post by ivan23178 »

LukeShu wrote: post #32
SrslyJosh wrote: post #29

It shouldn't be this hard to stop a service gracefully. This is far, far more complicated than SysV init, where you just need to drop a script into /etc/init.d symlink it from the appropriate rc directories. (For shutdown/reboot, you'd create symlinks in rc5.d and rc6.d named KNNwhatever, where NN is an integer that specifies the order the script will be run in. The "K" stands for "kill".) Edit: Note that my example…

> It shouldn't be this hard to stop a service gracefully. It's not. > you just need to drop a script into /etc/init.d symlink it from the appropriate rc directories. You just need to drop a unit file into /etc/systemd/system/ and symlink it from the appropriate /etc/systemd/system/${target}.wants/ directories. Don't tell me that "shutdown.target.wants" and "reboot.target.wants" are harder than "rc0.d" and "rc6.d". A…

There's also middle ground. OpenRC doesn't need those weird numbers while still being simple sysvinit/bash scripts-based system (it has a simple dependency system too). I'm not completely sure about `shutdown` level scripts having access to network though (too lazy to check), but it's still worth mentioning here.

Re: Writing systemd units that stop gracefully before shutdown

#44

Post by empthought »

kentonv wrote: post #42
empthought wrote: post #13

Earlier quoted context omitted.

This is a weird take; most systems in data centers don’t have people walking from rack to rack yanking power cords, and most consumer systems don’t even have a power cord to yank.

Weird take? It is a table stakes expectation for most servers that they will not lose data when the power goes out, or when the kernel panics, or when the server itself crashes or runs out of memory. If your software requires graceful shutdown, that seems to imply that it will lose data in all those cases. You can perhaps use graceful shutdown to perform some optimization that allows subsequent startup to go faster,…

That software should not “even expect” a graceful shutdown is the weird take.

Servers can and do “lose data” all the time when they’re shut down unexpectedly. I don’t know why you’d think they don’t. If the data has been read from somewhere (a socket maybe) and not fsynced, it’ll be lost. I agree that the system needs to be designed in such a way that this is a recoverable state, but I disagree with the ideas that applications should not have a mode where buffers can accumulate for some period of time without being fsynced, and that there should be no attention paid to the common case, which is planned process stops (aka SIGTERMs) for a variety of reasons. System shutdown just being one of those reasons.

Re: Writing systemd units that stop gracefully before shutdown

#46

Post by LukeShu »

ivan23178 wrote: post #43
LukeShu wrote: post #32

Earlier quoted context omitted.

> It shouldn't be this hard to stop a service gracefully. It's not. > you just need to drop a script into /etc/init.d symlink it from the appropriate rc directories. You just need to drop a unit file into /etc/systemd/system/ and symlink it from the appropriate /etc/systemd/system/${target}.wants/ directories. Don't tell me that "shutdown.target.wants" and "reboot.target.wants" are harder than "rc0.d" and "rc6.d". A…

There's also middle ground. OpenRC doesn't need those weird numbers while still being simple sysvinit/bash scripts-based system (it has a simple dependency system too). I'm not completely sure about `shutdown` level scripts having access to network though (too lazy to check), but it's still worth mentioning here.

OpenRC's great! But it doesn't detract from the point that systemd isn't nearly as complicated as some people act like it is.

PS: Who's still using OpenRC with heavy ol' sysvinit instead of the lighter openrc-init?

Re: Writing systemd units that stop gracefully before shutdown

#47

Post by lmz »

empthought wrote: post #44
kentonv wrote: post #42

Earlier quoted context omitted.

Weird take? It is a table stakes expectation for most servers that they will not lose data when the power goes out, or when the kernel panics, or when the server itself crashes or runs out of memory. If your software requires graceful shutdown, that seems to imply that it will lose data in all those cases. You can perhaps use graceful shutdown to perform some optimization that allows subsequent startup to go faster,…

That software should not “even expect” a graceful shutdown is the weird take. Servers can and do “lose data” all the time when they’re shut down unexpectedly. I don’t know why you’d think they don’t. If the data has been read from somewhere (a socket maybe) and not fsynced, it’ll be lost. I agree that the system needs to be designed in such a way that this is a recoverable state, but I disagree with the ideas that ap…

I think in that situation they expect you to not ack the data before you called fsync (which is the same expectation most people have of their SQL database). Then the remote end can retry the operation.

Re: Writing systemd units that stop gracefully before shutdown

#48

Post by comex »

empthought wrote: post #13
SoftTalker wrote: post #11

I fundamentally disagree with the idea that software should require or even expect a graceful shutdown. You can never stop the user from yanking the power cord out of the socket, which is what they will do if you force a bunch of housekeeping to happen before shutdown. You have to deal with crash/power failure recovery anyway. So do your housekeeping on startup. Shutdown should be a quick and simple termination.

This is a weird take; most systems in data centers don’t have people walking from rack to rack yanking power cords, and most consumer systems don’t even have a power cord to yank.

> and most consumer systems don’t even have a power cord to yank.

Some do. And the rest occasionally forcibly reboot (kernel panic or hardware failure), need to be manually forcibly rebooted (due to frozen UI), or unexpectedly loose battery, all leading to the same outcome. At least, that’s been my experience with just about every computer, phone, tablet, smartwatch, game console, and smart TV I’ve ever owned. Plus a number of routers. Is your experience different?

Re: Writing systemd units that stop gracefully before shutdown

#49

Post by rfmoz »

The macOS init manager, LaunchD, doesn't offer an easy way to execute an script at shutdown.

By definition, it sends a SIGTERM signal to all of the daemons that it started. But as the script isn't started before, and doesn't keep a running PID, you don't have a clean way to do it.

I don't understand why they only implemented the SIGTERM call without any alternative.

Re: Writing systemd units that stop gracefully before shutdown

#50

Post by code_biologist »

SoftTalker wrote: post #11

I fundamentally disagree with the idea that software should require or even expect a graceful shutdown. You can never stop the user from yanking the power cord out of the socket, which is what they will do if you force a bunch of housekeeping to happen before shutdown. You have to deal with crash/power failure recovery anyway. So do your housekeeping on startup. Shutdown should be a quick and simple termination.

100%

https://en.wikipedia.org/wiki/Crash-only_software

Post reply on HN