Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why systemd is a problem for embedded Linux

kevinboone.me

Post reply on HN
Print view View original

281–290 of 308 posts

Re: Why systemd is a problem for embedded Linux

#281

Post by jalgos_eminator »

cyberax wrote: post #278
jalgos_eminator wrote: post #276

Earlier quoted context omitted.

OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices. > And OpenWRT abandoned this goal. They have not. Individual devices get too old to be supported, but there are plenty of 16 MB devices they still support. Its true that basically every device now has 32MB or more of flash, but there are still plenty out…

> OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices. procd is only slightly better than sysvinit, it has the same ideas, and only adds simple and incorrectly implemented event triggers. And yes, I spent days debugging issues that it caused. E.g. a USB drive for the logs becomes slow and changes the startu…

> So you're thinking about more and more corner cases, that require the stars to be just right for the size of systemd to matter.

These weren't corner cases, they were everyday reality. We had to intentionally slim down the firmware size to get it to fit comfortably since we added quite a bit of application code. Like I said before, we got a recovery firmware down to ~3MB, and the normal firmware (without our application) was ~8MB. Every megabyte counts at that point, so having a 1MB init system would be a non-starter. We were buying commodity hardware, so we did lots of shopping and there were still plenty of 16MB APs on the market 10 years ago.

You're right its not a problem if you buy new hardware, but OpenWRT is used to support older devices with limited hardware specs. Its one of the reasons people install it, to give new life to old hardware (rather than being forced to buy a new device).

By the way, I use and like systemd on full size computers. I actually know of a product that runs a full rpm based distro (with systemd) on an embedded device and it works great. But systemd isn't one size fits all.

Re: Why systemd is a problem for embedded Linux

#282

Post by RandomThoughts3 »

AnthonyMouse wrote: post #157
richardfey wrote: post #145

Earlier quoted context omitted.

My most common issues with systemd are related to those long timeouts when something at boot/shutdown is not working as intended, and unexplained/unexplainable changes to the order of boot of some components. For the former I have given up playing whackamole with all the timeouts you need to reconfigure, for the latter I didn't even try because I know that there's something peculiar about my setup that will never wor…

> now I have to avoid that and minimise tinkering/hacking. I think this right here hits at the crux of the issue. There are people who like systemd because it's integration-tested with itself and its own defaults, so if you never change those defaults you don't have many problems. Then there are people who don't like systemd because if you do have to change any of its defaults, it often doesn't go well. And, of cours…

> it often doesn't go well

Plenty of people changes systemd configuration all the time and it just goes fine. You live in fantasy.

Even op is basically saying: “my issue with systemd is that I dislike the timeout configuration of some services but I stubbornly refuse to change these configurable timeout durations because it would show that the problem was myself and I prefer blaming systemd.”

It takes no time whatsoever to get a boot graph with each services name and starting time. That’s an actual feature documented in the manual of systemd which solves OP issue. But of course it would require actually understanding something new and everything new is bad, isn’t it?

Re: Why systemd is a problem for embedded Linux

#283

Post by RandomThoughts3 »

strken wrote: post #57
rjzzleep wrote: post #29

Earlier quoted context omitted.

Random example. They randomly broke suspect-then-hibernate then the community manager gaslit people into trying to make them believe they don’t actually need the feature in the way it’s used and then silently acknowledged that their fix is broken after all. This is a common pattern by the way. But easily the most irritating thing for me is the simple issue that you used to be able to just go to /var/log/lastlog to se…

The linked GitHub issue is one of the most aggravating exchanges I've ever read: https://github.com/systemd/systemd/issues/25269 . I don't understand how you can read someone telling you that they want to suspend and then hibernate after a set duration, but A) not understand why this would be desirable, B) not understand that this is compatible with also hibernating at low battery, and C) not understand your own lack…

> The linked GitHub issue is one of the most aggravating exchanges

Hyperbolic much?

Because if you actually read the whole issue, here we have one maintainer acting out of line of refusing to process the issue until another one (Yu Watanabe) interjects, says there is an issue and actually commit a patchset adding the option asked for.

Re: Why systemd is a problem for embedded Linux

#284

Post by RandomThoughts3 »

lmm wrote: post #269
vetinari wrote: post #243

Earlier quoted context omitted.

`systemctl disable` and `systemctl mask` work today just like they worked decade and half ago.

I don't know if they're the recommended way of doing things today, but they weren't what was in the manual when I was learning.

They are there from the start and were always the recommended way to do things in the documentation. I mean the command is literally disable.

The command to list services enabled on boot is also completely esoteric: systemctl list-unit-files --state=enabled

Re: Why systemd is a problem for embedded Linux

#285

Post by oblio »

AnthonyMouse wrote: post #272
oblio wrote: post #261

Earlier quoted context omitted.

> And, of course, the latter behavior as a box users are expected to live in is poisonous, because everyone is being conditioned to be passive and uniform. No, it's not, this assumes that the other camp are all idiots. Mainstream distros should be rock solid and boring. Linux never got anywhere with the standard distros because they're all so different. Tinkering is a different mindset (and has a different place) tha…

> Linux never got anywhere with the standard distros because they're all so different. This is eliding the difference in what you want to be standardized. Take system logging for example. You definitely want some standard interface to do it so all the different daemons and things can implement it once regardless of which system logger the distribution is using. But once it passes that data to the other program, it's…

We've had Unix for what, close to 55 years now? Nobody standardized and got widespread adoption for those standards, which BTW, especially when you look at corporate interests regarding the web, are easily sabotaged.

So I'd rather have the epoxy open source standard than no standard.

Re: Why systemd is a problem for embedded Linux

#286

Post by blueflow »

helloka wrote: post #266
bmicraft wrote: post #230

Earlier quoted context omitted.

Ubuntu is literally trying to replace it with dracut as we speak.

https://dracut-ng.github.io/dracut-ng/developer/compatabilit... Dracut is used both in Void Linux and on Alpine without systemd and with busybox. It even runs continuous integration with musl based containers.

Alpine uses mkinitfs per default, tho.

Re: Why systemd is a problem for embedded Linux

#287

Post by blueflow »

chasil wrote: post #249
blueflow wrote: post #51

Earlier quoted context omitted.

That "occasional weird problem" is because systemd is not designed to be used with other software. While you technically still have the choice to roll your own non-systemd initramfs, it will be an uphill battle.

The embedded people will naturally look at busybox. I saw it running on a credit card scanner at Old Navy a few years ago. It has an init: $ /home/busybox-1.35 init --help BusyBox v1.35.0 (2022-01-17 19:57:02 CET) multi-call binary. Usage: init Init is the first process started during boot. It never exits. It (re)spawns children according to /etc/inittab. Signals: HUP: reload /etc/inittab TSTP: stop respawning until…

This is what Alpine Linux use, Alpine also uses OpenRC for service startup.

Re: Why systemd is a problem for embedded Linux

#288

Post by strken »

RandomThoughts3 wrote: post #283
strken wrote: post #57

Earlier quoted context omitted.

The linked GitHub issue is one of the most aggravating exchanges I've ever read: https://github.com/systemd/systemd/issues/25269 . I don't understand how you can read someone telling you that they want to suspend and then hibernate after a set duration, but A) not understand why this would be desirable, B) not understand that this is compatible with also hibernating at low battery, and C) not understand your own lack…

> The linked GitHub issue is one of the most aggravating exchanges Hyperbolic much? Because if you actually read the whole issue, here we have one maintainer acting out of line of refusing to process the issue until another one (Yu Watanabe) interjects, says there is an issue and actually commit a patchset adding the option asked for.

I cannot think of any exchange I've read that aggravated me more than this one.

I'm hedging my bets with "one of" because there might be something I've forgotten, but that specific series of events is like reading a transcript of The Trial if it happened in real life. It is seriously extremely annoying to me and I'm both intrigued and concerned that it doesn't even rank in your list.

Re: Why systemd is a problem for embedded Linux

#289

Post by crote »

almatabata wrote: post #268
crote wrote: post #216

Earlier quoted context omitted.

At quantities of 100, 512Mbit of RAM is $1.01 [0]. 1Gbit of RAM is $1.10 [1]. 2Gbit is $1.05 [2]. 4Gbit is $1.16 [3]. It is only at 8Gbit that prices substantially increase to $2.30 [4]. So no, at those sizes price really does not change all that much, and installing 512MB of RAM instead of 64MB only increases the product's cost by $0.15. It's a commodity made on legacy processes, things like packaging, testing, and…

At a company I worked at they explicitly told us they will do anything to avoid upgrading the hardware from 1GB to 4GB because it increases costs. They would rather we optimize the software to use less RAM than upgrade the hardware. I remember arguing as well with people about 0.10$ components. They told me it was a no go, not even a point of bringing it up. Sometimes even a 0.01$ is a big deal.

Yeah, prices do indeed go up beyond 1GB - but we're talking about systemd needing 8MB of RAM. With small RAM chips there is more variation between parts than there is between sizes - hence the linked 2Gbit chip being cheaper than the 1Gbit one despite both of them being the cheapest option at LCSC. Those 8MBytes of systemd might push you from needing 1Gbit of RAM to 2Gbit, but it isn't going to push you from 8Gbit to 64Gbit - and as shown 1Gbit and 2Gbit don't meaningfully differ in price.

There are a lot of other factors involved with respinning hardware which make an upgrade a lot more expensive than simply a BOM increase. I can definitely understand why an existing product wouldn't be upgraded, but for a new product going to a bigger memory chip is a far smaller hurdle. The added software engineering time for working around RAM limitations can easily outweigh any money saved on the BOM, with choosing a smaller chip ending up being penny-wise pound-foolish.

And indeed, an extra $0.10 or even $0.01 can be a big deal. But those cheap systems usually aren't powerful enough to meaningfully run Linux in the first place: just because you can technically hack a $1.00 RP2040 or ESP32 into running Linux doesn't mean it is a good idea to actually do so. If your product is both cheap enough that it represents a significant fraction of your BOM and high-volume enough that you can afford the engineering time in the first place, why not use a purpose-built embedded OS like Zephyr?

Re: Why systemd is a problem for embedded Linux

#290

Post by richardfey »

RandomThoughts3 wrote: post #282
AnthonyMouse wrote: post #157

Earlier quoted context omitted.

> now I have to avoid that and minimise tinkering/hacking. I think this right here hits at the crux of the issue. There are people who like systemd because it's integration-tested with itself and its own defaults, so if you never change those defaults you don't have many problems. Then there are people who don't like systemd because if you do have to change any of its defaults, it often doesn't go well. And, of cours…

> it often doesn't go well Plenty of people changes systemd configuration all the time and it just goes fine. You live in fantasy. Even op is basically saying: “my issue with systemd is that I dislike the timeout configuration of some services but I stubbornly refuse to change these configurable timeout durations because it would show that the problem was myself and I prefer blaming systemd.” It takes no time whatsoe…

> Plenty of people changes systemd configuration all the time and it just goes fine. You live in fantasy.

"since I have never experienced what you say, it must be fantasy"

> Even op is basically saying: “my issue with systemd is that I dislike the timeout configuration of some services but I stubbornly refuse to change these configurable timeout durations because it would show that the problem was myself and I prefer blaming systemd.”

This is a perfect example of toxicity; I have been successfully using systemd for years and I am entitled to point out what I dislike, I do not have to love everything of it, it's not a religion nor a cult. Your reply tells more about yourself than the topic of the discussion at hand.

> It takes no time whatsoever to get a boot graph with each services name and starting time. That’s an actual feature documented in the manual of systemd which solves OP issue. But of course it would require actually understanding something new and everything new is bad, isn’t it?

You're missing the point, the problem is not changing timeouts but preventing failure and achieving an overall deterministic behaviour out of your system, without ignoring failures. But I refuse further eating these baits, you seem more interested in creating some flames rather than having constructive discussions.

Post reply on HN