21–30 of 83 posts
I've always preferred daemontools and runit's ideology here. If a service dies, wait one second, then try starting it. Do this forever. The last thing I need is emergent behavior out of my service manager.
I’ve been bitten by the restart limit many times. Our application server (backend) was crash looping, newest build fixed the crash, but systemd refused to restart the service due to the limit. A subtle but very annoying default behavior.
> I would guess the developers wanted to prevent laptops running out of battery too quickly And I would guess sysadmins also don't like their logging facilities filling the disks just because a service is stuck in a start loop. There are many reasons to think a service failing to start multiple times in a row won't start. Misconfiguration is probably the most frequent reason for that.
Heh. We used syslog at one place, with it configured to push logs into ELK. The ingestion into ELK broke … which caused syslog to start logging that it couldn't forward logs. Now that might seem like screaming into a void, but that log went to local disk, and syslog retried it as fast as disk would otherwise allow, so instantly every machine in the fleet started filling up its disks with logs. (You can guess how we n…
> Why does systemd give up by default? > I’m not sure. If I had to speculate, I would guess the developers wanted to prevent laptops running out of battery too quickly because one CPU core is permanently busy just restarting some service that’s crashing in a tight loop. sigh … bounded randomized exponential backoff retry. (exponential: double the maximum time you might wait each iteration. Randomized: the time you wa…
I've always preferred daemontools and runit's ideology here. If a service dies, wait one second, then try starting it. Do this forever. The last thing I need is emergent behavior out of my service manager.
Systemd can do that exactly that. it just doesn't do that by default. But if that's what you want, it's trivial
I’ve been bitten by the restart limit many times. Our application server (backend) was crash looping, newest build fixed the crash, but systemd refused to restart the service due to the limit. A subtle but very annoying default behavior.
are you saying systemd was refusing to restart after manual intervention?
I’ve been bitten by the restart limit many times. Our application server (backend) was crash looping, newest build fixed the crash, but systemd refused to restart the service due to the limit. A subtle but very annoying default behavior.
Did your deployment process/script not include restarting the service?
It would be nice if `RestartSec` weren't constant. Then you could have the default be 100ms for one-time blips, but (after a burst of failures) fall back gradually to 10s to avoid spinning during longer outages. That said, beware of failure chains causing the interval to add up. AFAIK there's no way to have the kernel notify you of when a different process starts listening on a port.
There's `RestartSteps` and `RestartMaxDelaySec` for that, see the manpage `systemd.service`.
Available since systemd 254, released July 2023 (only 1 release since then). Huh, has release rate severely slowed down?
This must be a different philosophy. When I see something like this happening, I investigate to find out why the service is failing to start, which usually uncovers some dependency that can be encoded in the service unit, or some bug in the service.
> Why does systemd give up by default? > I’m not sure. If I had to speculate, I would guess the developers wanted to prevent laptops running out of battery too quickly because one CPU core is permanently busy just restarting some service that’s crashing in a tight loop. sigh … bounded randomized exponential backoff retry. (exponential: double the maximum time you might wait each iteration. Randomized: the time you wa…
Especially that service startup failure is usually not something that gets fixed on its own, like a network connection (where exponential backoff is (in)famous). A bad config file, or a failed disk won’t recover in 10 minutes on its own, so systemd’s default makes sense here, I believe.