Live data from Hacker News

Why systemd is a problem for embedded Linux

kevinboone.me

101–110 of 308 posts

Re: Why systemd is a problem for embedded Linux

#101

I’ve never used it as an init replacement, but runit ( https://smarden.org/runit/runit.8 ) is great for supervising applications and services. - It’s built into BusyBox and is very lightweight - Configuration is dead simple (everything is basically shell scripts in the filesystem; no special config language) - Includes logging functionality with rotation - Easily controllable from other applications using named pipes…

For what it's worth, runit is the default on Void Linux.

Re: Why systemd is a problem for embedded Linux

#102
post #72

Earlier quoted context omitted.

systemd itself is also large in storage size(8MB?), which is why openwrt has its own procd replacements, and yocto is not default to systemd either due to this. additionally, systemd does not work well with musl which is very popular in embedded linux.

systemd binary is around 1.5Mb, with the shared libsystemd it's around 2.5Mb. Both can be reduced by not including compressors (the infamous libz), audit libraries, SELinux, etc. The minimum binaries size that can be obtained by trivial removals is around 2Mb. OpenWRT should just stop being stubborn and integrate it. They dropped support for 4Mb devices loooooong ago, and recently bumped the minimum specs to 16Mb. >…

> OpenWRT should just stop being stubborn and integrate it.

Why? If it fails to provide benefits commensurate to the effort required to integrate it, then there's no reason to do the work.

OpenWRT is for routers and access points... devices which usually have a fixed [hardware] configuration, and very little need for the absurdly-complex (and in my professional experience, often subtly buggy) event-driven systems that systemd provides. "Every system everywhere gets to use the same unit files" isn't a significant benefit in practice... EVERYTHING that needs nontrivial pre-/post- start/stop verification and/or configuration has systemd run a shell script (or other such helper) anyway... so you only get to just write a unit file and call it a day in the most trivial of cases.

Re: Why systemd is a problem for embedded Linux

#103

Wait so systemd is a problem for embedded Linux because... it uses 250MB of RAM? So it's only really a problem for systems where using a mainstream distro probably wouldn't even be a consideration anyway (yocto seems to be popular, as mentioned above). I'm not seeing a strong argument here. At the risk of inviting a parade of "actually, one time at band camp I saved the day with a first-gen raspberry pi zero", how ma…

(Almost) All the routers that people have in their homes - the ones running OpenWRT - have between 64-256 MB RAM, and no swap. Raspberry Pis are not "embedded", they are cheap, low power consumer devices.

Re: Why systemd is a problem for embedded Linux

#104
I have no philosophy issue with systemd. The biggest show-stopper (on embedded Linux) for me is its install size. We have edge devices with only 64M RAM and 128M NAND flash. For an OTA update (via Bluetooth sometimes) only ~20M in total, adding ~5M systemd plus its dependencies doesn't make sense. 5MiB is roughly equivalent to the entire Linux kernel image on those devices.

Among the components of systemd, the part that has the lowest added value for our use is dbus. It's designed primarily for desktops and useless for most of our systems, but unfortunately, it is also a hard requirement for systemd.

Re: Why systemd is a problem for embedded Linux

#105
post #17

For me as a user, systemd is far better than old init systems. The logging alone is worth it to switch to it. Add more ram to the embedded board and welcome to the future.

> The logging alone is worth it to switch to it.

Weird. In my professional life, I've found systemd's logging to be woefully lacking when compared to systems that just run every daemon through syslog (or -even more retro- "each daemon writes stdout & stderr to disk, which then gets sent to your log sink of choice").

What I usually find is that a ton of information never, ever makes it into journalctl, so I always need to go to the actual log files on disk. [0]

On top of that, journalctl just fucking giving up when logs have been rotated is totally bogus. (If this has been fixed, then the fix hasn't made its way to the systems I work with.)

ALSO, have they fixed the issue where minor data corruption just totally fucks your log file to death... rather than what happens with regular log files and you get an unreadable section that's bracketed by good data?

[0] Yes, you could reasonably say "Well, those services are not correctly configured to log to journalctl!". But my point here is that I see this happen so often that journalctl is like my tenth stop for log messages, rather than my first.

Re: Why systemd is a problem for embedded Linux

#106

There is another aspect to the memory footprint: Does systemd fit into the L1 cache? Does sysvinit? Both x86 and Linux (page cache) rely on caches for performance.

I'm not sure that fitting a init system in L1 cache should be a priority when it would be swapped out almost immediately on doing further work, aka the purpose of the device.

Re: Why systemd is a problem for embedded Linux

#107

I couldn't disagree more: I've worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience. I won't pretend there aren't occasional weird problems... but there's always a solution, here's a recent example: https://github.com/systemd/systemd/issues/34683 Memory use is irrelevant to me: every embedded Linu…

> every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM That is almost by definition not an embedded device. There's a reason we have vfork().

Well, you can gatekeep all you want, but it's increasingly practical and common to have what would have seemed like an absurd amount of RAM a decade ago on things like toasters.

Re: Why systemd is a problem for embedded Linux

#108
post #72

Earlier quoted context omitted.

systemd itself is also large in storage size(8MB?), which is why openwrt has its own procd replacements, and yocto is not default to systemd either due to this. additionally, systemd does not work well with musl which is very popular in embedded linux.

systemd binary is around 1.5Mb, with the shared libsystemd it's around 2.5Mb. Both can be reduced by not including compressors (the infamous libz), audit libraries, SELinux, etc. The minimum binaries size that can be obtained by trivial removals is around 2Mb. OpenWRT should just stop being stubborn and integrate it. They dropped support for 4Mb devices loooooong ago, and recently bumped the minimum specs to 16Mb. >…

OpenWRT is VERY sensitive to firmware size. I developed for a device with 16 MB flash that had both a normal (~11 MB) and recovery firmware (~3MB) on flash. The rest of flash was used for persistent storage, bootloaders, and calibration data. This is very common for older routers and APs which is the whole reason that OpenWRT exists.

Re: Why systemd is a problem for embedded Linux

#109

Systemd, Wayland, DBus, and other modern Linux "solutions" are all problematic for the same reason: they are complex monoliths that are extremely difficult to replace, in part or in whole. Systems often won't work without them, because the entire ecosystem has grown dependent on them. You literally can't run a modern desktop without a bunch of "shims" whose sole purpose is to fake being systemd. The best designs, sad…

Where have you heard that Slackware will be using systemd next release? I have been using slackware for years, and this is news to me.

Re: Why systemd is a problem for embedded Linux

#110

I couldn't disagree more: I've worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience. I won't pretend there aren't occasional weird problems... but there's always a solution, here's a recent example: https://github.com/systemd/systemd/issues/34683 Memory use is irrelevant to me: every embedded Linu…

> every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM. If I'm on a tiny machine where I care about 8MB RSS, I'm not running Linux, I'm running Zypher or FreeRTOS

The gap between “over 1GB of RAM” and 8MB RSS contains the vast majority of embedded Linux devices.

I, too, enjoy when the RAM budget is over 1GB. The majority of cost constrained products don’t allow that, though.

That’s said, it’s more than just RAM. It increases boot times (mentioned in the article) which is a pretty big deal on certain consumer products that aren’t always powered on. The article makes some good points that you’ve waved away because you’ve been working on a different category of devices.

Post reply on HN