Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systemd: Enable indefinite service restarts

michael.stapelberg.ch

Post reply on HN
Print view View original

11–20 of 83 posts

Re: Systemd: Enable indefinite service restarts

#11

Post by dijit »

o11c wrote: post #10

It would be nice if `RestartSec` weren't constant. Then you could have the default be 100ms for one-time blips, but (after a burst of failures) fall back gradually to 10s to avoid spinning during longer outages. That said, beware of failure chains causing the interval to add up. AFAIK there's no way to have the kernel notify you of when a different process starts listening on a port.

> AFAIK there's no way to have the kernel notify you of when a different process starts listening on a port.

You can use mandatory access control for this.

AppArmour or SELinux are examples.

Unfortunately they are hard, not sexy and sysadmins (people who tend to do not sexy hard things) are a dead/dying breed

Re: Systemd: Enable indefinite service restarts

#12

Post by dijit »

mise_en_place wrote: post #9

I’ve been bitten by the restart limit many times. Our application server (backend) was crash looping, newest build fixed the crash, but systemd refused to restart the service due to the limit. A subtle but very annoying default behavior.

are you saying systemd was refusing to restart after manual intervention?

Re: Systemd: Enable indefinite service restarts

#14

Post by nomel »

o11c wrote: post #10

It would be nice if `RestartSec` weren't constant. Then you could have the default be 100ms for one-time blips, but (after a burst of failures) fall back gradually to 10s to avoid spinning during longer outages. That said, beware of failure chains causing the interval to add up. AFAIK there's no way to have the kernel notify you of when a different process starts listening on a port.

> AFAIK there's no way to have the kernel notify you of when a different process starts listening on a port.

Would the ExecCondition be appropriate here, minimally, with a script that runs `lsof -nP -iTCP:${yourport} -sTCP:LISTEN`?

Re: Systemd: Enable indefinite service restarts

#15

Post by saint_yossarian »

o11c wrote: post #10

It would be nice if `RestartSec` weren't constant. Then you could have the default be 100ms for one-time blips, but (after a burst of failures) fall back gradually to 10s to avoid spinning during longer outages. That said, beware of failure chains causing the interval to add up. AFAIK there's no way to have the kernel notify you of when a different process starts listening on a port.

There's `RestartSteps` and `RestartMaxDelaySec` for that, see the manpage `systemd.service`.

Re: Systemd: Enable indefinite service restarts

#16

Post by ElectricSpoon »

> I would guess the developers wanted to prevent laptops running out of battery too quickly

And I would guess sysadmins also don't like their logging facilities filling the disks just because a service is stuck in a start loop. There are many reasons to think a service failing to start multiple times in a row won't start. Misconfiguration is probably the most frequent reason for that.

Re: Systemd: Enable indefinite service restarts

#17

Post by twic »

ElectricSpoon wrote: post #16

> I would guess the developers wanted to prevent laptops running out of battery too quickly And I would guess sysadmins also don't like their logging facilities filling the disks just because a service is stuck in a start loop. There are many reasons to think a service failing to start multiple times in a row won't start. Misconfiguration is probably the most frequent reason for that.

Exactly. If a service crashes within a second ten times in a row, it's not going to come up cleanly an eleventh time. The right thing to do is stay down, and let monitoring get the attention of a human operator who can figure out what the problem is. Continually rebooting is just going to fill up logs, spam other services, and generally make trouble.

I'm sure there are exceptions to this. For those, set Restart=always. But it's an absolutely terrible default.

Re: Systemd: Enable indefinite service restarts

#18

Post by deathanatos »

> Why does systemd give up by default?

> I’m not sure. If I had to speculate, I would guess the developers wanted to prevent laptops running out of battery too quickly because one CPU core is permanently busy just restarting some service that’s crashing in a tight loop.

sigh … bounded randomized exponential backoff retry.

(exponential: double the maximum time you might wait each iteration. Randomized: the time you want is a random amount, between [0, current maximum] (yes, zero.). Bounded: you stop doubling at a certain point, like 5 minutes, so that we'll never wait longer than 5 minutes; otherwise, at some point you're waiting for ∞s, which I guess is like giving up.)

(The concern about logs filling up is a worse one. It won't directly solve this, but a high enough max wait usually slows the rate of log generation enough that it becomes small enough to not matter. Also do your log rotations on size.)

Re: Systemd: Enable indefinite service restarts

#19

Post by deathanatos »

ElectricSpoon wrote: post #16

> I would guess the developers wanted to prevent laptops running out of battery too quickly And I would guess sysadmins also don't like their logging facilities filling the disks just because a service is stuck in a start loop. There are many reasons to think a service failing to start multiple times in a row won't start. Misconfiguration is probably the most frequent reason for that.

Heh. We used syslog at one place, with it configured to push logs into ELK. The ingestion into ELK broke … which caused syslog to start logging that it couldn't forward logs. Now that might seem like screaming into a void, but that log went to local disk, and syslog retried it as fast as disk would otherwise allow, so instantly every machine in the fleet started filling up its disks with logs.

(You can guess how we noticed the problem…)

Also logrotate. (And bounded on size.)

Re: Systemd: Enable indefinite service restarts

#20

Post by franknord23 »

I believe this allows you to have cascading restart strategies, similar to what can be done in Erlang/OTP: Only after the StartLimit= has been reached, systemd considers the service as failed. Then services that have Required= set on the failed service will be restarted/marked failed as well.

I think you can even have systemd reboot or move the system into a recovery mode (target) if an essential unit does not come up. That way, you can get pretty robust systems that are highly tolerant to failures.

(Now after reading `man systemd.unit`, i am not fully sure how exactly restarts are cascaded to requiring units.)

Post reply on HN