Why Did ArchLinux Embrace Systemd? (2016)

Until very recently having wrong permission on one of the networkd config files would result in networkd ignoring the rest of the perfectly readable network configuration files too, cutting off all network connectivity. (Systemd also requested the permission change iself.)

Currently there's a bug where if you have a bridge and all interfaces are disconnected from the bridge (I use hot-pluggable USB network interfaces) when you re-connect them, networkd will keep the bridge off. If you restart systemd-networkd it will fix itself.

I can name at least two instances already where I lost networking in my entire SBC farm, and had to pop out all microSD cards and fix the breakage manually, just because of systemd-networkd bugs. And I don't do anything crazy, just a single ethernet interface and two overlay wireguard networks, and mostly default config.

And fixing it yourself is not easy. I can do C coding perfectly fine, but I run 3 different CPU archs, so fixing it myself means compiling (3 times) and distributing my own systemd version to all my machines until upstream fixes the bug. It's not pleasant, nor easy.

I certainly understand people who miss having an ability to more readily script the system startup/network config.

Regardless, every time these types of threads come about I feel like I _have_ to speak because there's a lot of bitterly divided people regarding this issue. (or, rather the people who are bitterly opposed and the people who don't care, think systemd is a net good in the vast majority of cases or those who think the backlash is very much unwarranted).

I'd like to say first off: SystemD solves real problems that nothing before it was solving, maybe it doesn't solve those issues in the way I personally like, but it definitely does and was one of the first to do so- so for this reason the original maintainers should be warranted respect, it's also a given that it's not the developers fault that this is so widely adopted or even forced. I see them as problem solvers first and foremost so any gripes I have with systemd as a thing which is foisted upon me is definitely not on them.

However, I do consider that systemd as a new -layer- between user land and kernel space (called "system space" by Poettering) necessitates a lot of breaking changes. I do not consider this by itself as a bad thing, but having a poorly defined interface for these things does not enable systemd to be replaced without some form of "systemd-esque" emulation in future.

Thus while we have improved from where we were, we do not have fertile ground to improve in the future.

Additionally; SystemD as a "modular" system is, in my opinion, a myth; even if you take for granted the fact that systemd+journald are the only core dependencies, you then quickly start to drag in more and more rapidly (udev and systemd-resolved being famous examples).

Some design paradigms are scary, for instance socket activation leaves PID1 listening for traffic on the network, by default bound to all interfaces. (a good example is to look at cockpit) this could be a very large footgun if there exists any form of exploit in systemd.. and since systemd is not written in a memory safe language, and the overwhelming majority of security bugs are memory safety bugs... this is mildly concerning.

There are other concerns regarding opacity of the dependency system, the fact that no-two-boots are the same and thus some race conditions are likely to never be solved by me. (a lowly sysadmin type who only knows enough C to make linked lists). The design very much reminds me of Microsoft Windows in many ways, but perhaps that was a good design to choose from?

FWIW: SystemD is amazing on my laptop, I am really happy with it. But for my servers I am very unhappy with it, but the major distributions chose to use systemd and I'm a sysadmin so I'm beholden to what the company considers stable and developers.

I very much begrudge the fact that I'm not allowed to have a dissenting opinion on this.

I also hate that when people talk about systemd vs "anything", invariably people assume I am talking about openRC or sysvinit, but there are much better init systems available today, such as Runit.

SystemD is not a panacea, it should leave fertile ground for replacement. If that cannot be done it should not be adopted because that is a scary amount of lock-in.

However, after trying SystemD for one week, I became convinced that the designers of SystemD are incompetent, so they cannot be trusted with a component of such importance for a computer.

I am normally a Gentoo user, but a few years ago I wanted to install Linux in a hurry on a small computer. With a slow CPU installing Gentoo can take several hours, unless you have a previously prepared image, so I decided to try ArchLinux.

The installation was indeed fast and without problems. However, SystemD behaved in an unexpected way. SystemD is advertised as booting quickly. It indeed booted quickly, but no faster than my optimized Gentoo systems.

On the other hand the shutdown with SystemD was very slow. This was a huge surprise, because never before and never after have I encountered any computer where the shutdown is annoyingly slow.

Moreover, sometimes the shutdown FAILED, which is also something that I have never encountered on any computer not using SystemD, and I have used thousands of computers, with many kinds of operating systems, including at least eight or nine UNIX flavors other than Linux.

The failure of shutdown was apparently due to some kind of race condition when some process was killed faster than some SystemD component expected and that SystemD component was still trying to communicate via DBUS with the defunct process and it blocked because its messages no longer reached the destination.

I suppose that this bug might have been corrected eventually, but this is a design error that I consider unforgivable, i.e. to conceive a shutdown sequence which depends on successful inter-process communication.

After seeing such a mistake I have lost any confidence in the abilities of the SystemD developers, so I have never tried it again.

SystemD is much the same way, I'm coloured by my impressions from Fedora21-22, where it.. didn't work great.

The issue is with poetterware, it's _incredibly_ difficult to discover /why/ it's not working properly, which is dangerous for an init system (or, an "everything between your application and kernel"-system), a failure rate of 0.0001% is still an unfathomably enormous number of systems; and due to how it works it lends itself to an incredible host of threading issues.