Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systemd: Enable indefinite service restarts

michael.stapelberg.ch

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Systemd: Enable indefinite service restarts

#51

Post by BarbaryCoast »

tadfisher wrote: post #3

This must be a different philosophy. When I see something like this happening, I investigate to find out why the service is failing to start, which usually uncovers some dependency that can be encoded in the service unit, or some bug in the service.

...and now you know why I don't run systemd. I believe their thought process is: what would Windows do? This is an example. For instance, the desktop shell still crashes often. In the old days, this would lock up the keyboard and mouse, and you'd have to power cycle. But MS "fixed" it by simply adding infinite restarts to the system. Now we have systemd. When something crashes, there's no need to fix the bug, just restart it.

My favorite new misfeature is PulseAudio. These geniuses actually built code for a multi-user, multi-tasking OS...which will only run for ONE user, and then only if that user is logged in. So forget running cron jobs, and sounding an alert if something needs attention.

This is all code produced by FreeDesktop[.]org. Thanks to them, your industrial strength, mission-critical server OS is now only suitable for single-user desktop systems.

Re: Systemd: Enable indefinite service restarts

#52

Post by rconti »

freedomben wrote: post #21
akira2501 wrote: post #8

I've always preferred daemontools and runit's ideology here. If a service dies, wait one second, then try starting it. Do this forever. The last thing I need is emergent behavior out of my service manager.

Systemd can do that exactly that. it just doesn't do that by default. But if that's what you want, it's trivial

... and how many of us knew this before the article?

Re: Systemd: Enable indefinite service restarts

#53

Post by vidarh »

franknord23 wrote: post #20

I believe this allows you to have cascading restart strategies, similar to what can be done in Erlang/OTP: Only after the StartLimit= has been reached, systemd considers the service as failed. Then services that have Required= set on the failed service will be restarted/marked failed as well. I think you can even have systemd reboot or move the system into a recovery mode (target) if an essential unit does not come u…

You can trigger units explicitly on failure with OnFailure=someservice as well (and since you can parameterize service names, you can have e.g. a single failure@.service that'll do whatever you prefer once a service fails.

OnFailure makes it easy to implement more complex restart or notification logic.

Re: Systemd: Enable indefinite service restarts

#54

Post by melolife »

ElectricSpoon wrote: post #16

> I would guess the developers wanted to prevent laptops running out of battery too quickly And I would guess sysadmins also don't like their logging facilities filling the disks just because a service is stuck in a start loop. There are many reasons to think a service failing to start multiple times in a row won't start. Misconfiguration is probably the most frequent reason for that.

I've seen bad service design having e.g.

Before=systemd-user-sessions.service

This means that as long as systemd is trying to (re)start the service, nobody can log in. Which is a problem with infinite restarts.

It's still pretty easy to accidentally set up an infinite restart loop with the default settings if your service takes more than 2s to crash.

Re: Systemd: Enable indefinite service restarts

#55

Post by vidarh »

dijit wrote: post #31
mise_en_place wrote: post #26

Earlier quoted context omitted.

Correct, because the startup limit had been reached: `service start request repeated too quickly, refusing to start`.

Thats terrifying, systemd shouldn't pretend to be smarter than manual intervention. That violates everything I ever enjoyed linux for, I left Windows because it thought it knew better than me.

Well, it's behaving as documented and following the semantics of your unit file.

Though I agree the default is poor, if it bites you more than once, write a standard template to reuse for your unit files and/or write a wrapper that calls reset-failed for you. Systemd is far from perfect, but this is a minor nuisance.

Re: Systemd: Enable indefinite service restarts

#56

Post by linsomniac »

__float wrote: post #40
bogota wrote: post #35

Earlier quoted context omitted.

The amount of times i had to fight and debug systemd compared to any other init system is at least 10x. Yes it does a lot of stuff for you and in others I had to write custom scripts but it was much more understandable and maintainable long term. Sadly systemd won and now i build my own OS without it.

Even for basic service running, complex dependencies are so much more manageable in systemd. I'm glad systemd "won"; it's much more maintainable IMO than shell scripts written once and forgotten about (until they break).

Agreed, I have some friends that don't like systemd, and I respect their opinions greatly, but my experience with systemd has been quite good over the last decade. I really like it much more than the old init.d scripts.

With one exception... I tried to set up a inetd-like service where a script was run on TCP connection with the output going out to the socket. It was tricky to get set up, but eventually I got it working... or so I thought. A few days later the system was super sluggish and I found that every connection that had been made to this service was left open and consuming resources. Switched that to an socat running under systemd and it's been fine since. Never could figure out what exactly the deal was.

Re: Systemd: Enable indefinite service restarts

#57

Post by vidarh »

mise_en_place wrote: post #29
tadfisher wrote: post #3

This must be a different philosophy. When I see something like this happening, I investigate to find out why the service is failing to start, which usually uncovers some dependency that can be encoded in the service unit, or some bug in the service.

That's exactly why systemd should blindly attempt to restart the service infinitely. Seperation of concerns. An init system should simply start and monitor services. That is what an init system is meant to do. The fact that systemd is overengineered and tries to do multiple things causes headaches for a lot of us. Busybox-init is one of the best alternatives, I would use that everywhere if I could.

It's trivial to make systemd do that if that is what you want, but there are also plenty of cases when that is not what you want and you then end up trying to write crash-proof startup scripts to provide backoff instead of just changing a flag in a unit file.

(And if you want a dumb unit system, there are plenty of options which will run just fine under systemd as a single unit so you never have to actually use systemd for your own services even if you're forced to use systemd for the overall system for whatever reason)

Re: Systemd: Enable indefinite service restarts

#58

Post by BobbyTables2 »

o11c wrote: post #10

It would be nice if `RestartSec` weren't constant. Then you could have the default be 100ms for one-time blips, but (after a burst of failures) fall back gradually to 10s to avoid spinning during longer outages. That said, beware of failure chains causing the interval to add up. AFAIK there's no way to have the kernel notify you of when a different process starts listening on a port.

> AFAIK there's no way to have the kernel notify you of when a different process starts listening on a port.

For startup, I’d argue the proper way is for the process to bind the socket before forking as a daemon.

With such a design, one can launch a list of dependent processes without worrying how long they each take to start up. No polling loops needed!

Of course, that requires some careful design — “fork and forget” is too appealing. The process would be responsible for creating its PID file after forking but the socket before…

Alternatively, an IPC notification could be used but would require some sort of standardization to be generally useful.

Re: Systemd: Enable indefinite service restarts

#59

Post by rendaw »

BenjiWiebe wrote: post #50
twic wrote: post #17

Earlier quoted context omitted.

Exactly. If a service crashes within a second ten times in a row, it's not going to come up cleanly an eleventh time. The right thing to do is stay down, and let monitoring get the attention of a human operator who can figure out what the problem is. Continually rebooting is just going to fill up logs, spam other services, and generally make trouble. I'm sure there are exceptions to this. For those, set Restart=alway…

It might actually, if a network connection is temporarily down.

Or a disk not attached yet. Or another service it depends on being slow to finish starting up.

Re: Systemd: Enable indefinite service restarts

#60

Post by otterley »

kaba0 wrote: post #30
deathanatos wrote: post #18

> Why does systemd give up by default? > I’m not sure. If I had to speculate, I would guess the developers wanted to prevent laptops running out of battery too quickly because one CPU core is permanently busy just restarting some service that’s crashing in a tight loop. sigh … bounded randomized exponential backoff retry. (exponential: double the maximum time you might wait each iteration. Randomized: the time you wa…

Arguably, this logic should live in another place that monitors the service. Especially that service startup failure is usually not something that gets fixed on its own, like a network connection (where exponential backoff is (in)famous). A bad config file, or a failed disk won’t recover in 10 minutes on its own, so systemd’s default makes sense here, I believe.

systemd is a service monitor. It wouldn't be nearly as useful if it wasn't!
Post reply on HN