Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why systemd is a problem for embedded Linux

kevinboone.me

Post reply on HN
Print view View original

141–150 of 308 posts

Re: Why systemd is a problem for embedded Linux

#141

Post by cyberax »

oguz-ismail wrote: post #86
m463 wrote: post #83

I think what he's running into is "the unix philosophy". It is basically about small tools, limited in scope, that solve a problem surgically and comprehensively. They work together by mixing and matching to solve the problem. systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing. I don't know, it is sort of like busybox, repla…

> systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing And it's slow. Windows 11 boots in 5 seconds on my laptop, Ubuntu 22 used to take about a minute before I finally made the switch.

My system boots systemd in 2 seconds from the firmware handover to my code fully running. And this includes device initialization and a DHCP lease renewal.

On VMs you can get to within 200ms for a full system.

You can use `systemd-analyze critical-chain` to visualize the boot process. The slowdown is likely caused by something unexpected.

Re: Why systemd is a problem for embedded Linux

#142

Post by BeefWellington »

metalforever wrote: post #99
detaro wrote: post #33

Earlier quoted context omitted.

> how many embedded systems in 2024 are really RAM-limited? A lot. Plenty embedded Linux stuff still ships with 256 MB or 512 MB RAM, and the wishlist for features in the software running on top is always getting longer than initially planned... But of course the average has moved up, and it's not unusual to see systemd in embedded systems either (while I don't have a number at hand, the 250MB number seems off to me)…

I remember when I used to run gnome 2 (i.e. mate) on a machine with 256mb of ram. It was a full experience and it worked with youtube videos and so on. What are we even doing .

Sure, back when videos were at most 480p/720p that's feasible.

I'm not saying software is any more efficiently written these days but I do think it's important to recognize that just the act of pushing more pixels on its own requires more RAM.

Re: Why systemd is a problem for embedded Linux

#143

Post by vbezhenar »

jcalvinowens wrote: post #116
Aurornis wrote: post #111

Earlier quoted context omitted.

You have been living in a strange world if you’ve been getting away with 1GB in the average consumer IoT device for the past 5 years. That’s not typical at all. I’ve done a lot of work with production runs in the millions. There is no way we’d put that much RAM on a device unless it was absolutely, undeniably required for core functionality.

I've asked in the past, and been told that a even a 2x-3x difference in the amount of RAM made such a negligible difference in cost it was decided to go with the larger amount. I frankly have a hard time understanding how that can be true... but I can't really imagine why they wouldn't be honest with me about it.

Even Apple produced laptops with 8 GB RAM just recently, which they sold for hundreds of dollars with huge margins (AFAIK). If you're going to produce something with $50 cost, 1GB RAM cost will be meaningful.

In my experience production people will eat your soul for a single resistor if they can cut costs on it.

Re: Why systemd is a problem for embedded Linux

#144

Post by cyberax »

simoncion wrote: post #137
cyberax wrote: post #113

Earlier quoted context omitted.

OpenWRT is a mess. It started as a small distro for routers, but it stopped being that long ago. Now it can host telephony services, full-blown Docker orchestration systems, etc. So you get all the pleasures of complex systems running on antiquated infrastructure. > so you only get to just write a unit file and call it a day in the most trivial of cases. What "start/stop verification"? Give examples, please. Systemd…

> Now it can host telephony services, full-blown Docker orchestration systems, etc. What features that systemd provides that aren't provided by openrc, runit, monit, openrc's init system, etc, etc, etc that are needed by software like Asterisk, Docker/runc, Kubernetes, and similar? If what OpenWRT is running works fine to bring that up, then systemd doesn't have any relevant compelling features that aren't provided b…

> What features that systemd provides that aren't provided by openrc, runit, monit, openrc's init system, etc, etc, etc that are needed by software like Asterisk, Docker/runc, Kubernetes, and similar?

Dependencies, watchdogs, start only when the given network interfaces are available, declarative configuration so that can be partially customized, etc.

> If what OpenWRT is running works fine

Unless you try to update it. And end up with merge conflicts in the /etc directory. And then it stops booting, and you have to get in via the serial port.

No, OpenWRT is most definitely not fine. It's stuck in the past, and doesn't support automated rollback, configuration management, etc.

> A concrete example of this is Gentoo's running of 'named-checkconf' in BIND 9's service file [0]

Wow, what a... But named config has always been a disaster zone.

With systemd _none_ of that crap is needed. It can use its own namespacing and cgroup support to ensure that BIND9 can't read outside of its allowlisted tree.

If you want to run /usr/bin/named-checkconf, then it's just an ExecStartPre directive in the unit file. And unlike SysV that just fails to run and forces you to dig into the logs, systemd provides you a nice status saying that the pre-exec had failed.

> You might look at that file and be like "wow, TL;WR", but do try to notice how much of that file is dedicated to creating useful, human-readable status messages and error messages with good corrective instructions.

Yeah, and hanging indefinitely, forcing me to do a trip to the datacenter at night (true story): https://lwn.net/Articles/619565/

Re: Why systemd is a problem for embedded Linux

#145

Post by richardfey »

cedilla wrote: post #80
fsckboy wrote: post #36

Earlier quoted context omitted.

> the good things are the things you don't notice; the things that just work sorry, no, at least in any meaningful sense, because init already just worked for me. and when something didn't, I could find how to fix it in a way that was transparent and I understood and could even modify or make better, without being connected to the internet looking for cargo cult incantations on stack overflow init was a tool; systemd…

Okay, I'll bite. What kind of problems do you have with the part of systemd that replicates sysvinit /every other day/?

My most common issues with systemd are related to those long timeouts when something at boot/shutdown is not working as intended, and unexplained/unexplainable changes to the order of boot of some components. For the former I have given up playing whackamole with all the timeouts you need to reconfigure, for the latter I didn't even try because I know that there's something peculiar about my setup that will never work nicely with systemd, there's simply not enough systems configured like that for upstream to care. I have accepted this new reality, but I know that before systemd I was able to fix any highly customised setup of mine, now I have to avoid that and minimise tinkering/hacking.

Re: Why systemd is a problem for embedded Linux

#147

Post by righthand »

samatman wrote: post #87
righthand wrote: post #55

Earlier quoted context omitted.

It is legal to sell unhealthy food, ie poisonous food. Sugar is a poison, drugs are poisons. You can legally sell escolar a fish that if eaten too much is poisonous. What is poisonous food? You can sell plastic that breaks down into the environment and people are drinking it in water, eating it in fish. Is that legally sold poisonous food?

Sugar is not, in fact, a poison. As for drugs: > Alle Dinge sind Gift, und nichts ist ohne Gift; allein die Dosis macht, dass ein Ding kein Gift ist. - Paracelsus, 1538 https://en.wikipedia.org/wiki/The_dose_makes_the_poison

Sugar is indeed poison.

Personally, I like to add a bit more science and reasoning to my understanding rather than rely on quotes from someone in 1538.

Re: Why systemd is a problem for embedded Linux

#148

Post by spockz »

kelnos wrote: post #125
jcalvinowens wrote: post #116

Earlier quoted context omitted.

I've asked in the past, and been told that a even a 2x-3x difference in the amount of RAM made such a negligible difference in cost it was decided to go with the larger amount. I frankly have a hard time understanding how that can be true... but I can't really imagine why they wouldn't be honest with me about it.

> I've asked in the past, and been told that a even a 2x-3x difference in the amount of RAM made such a negligible difference in cost it was decided to go with the larger amount That doesn't pass the sniff test. Look at retail RAM prices. Certainly the magnitude of the price is quite different than buying individual RAM chips at quantity, but the costs do scale up as RAM size goes up. Hell, look at RAM chip prices: y…

The retail price of a finished product has very little to do with the cost of individual components and more with profit margins or customer segmentation.

Re: Why systemd is a problem for embedded Linux

#149

Post by cyberax »

simoncion wrote: post #137
cyberax wrote: post #113

Earlier quoted context omitted.

OpenWRT is a mess. It started as a small distro for routers, but it stopped being that long ago. Now it can host telephony services, full-blown Docker orchestration systems, etc. So you get all the pleasures of complex systems running on antiquated infrastructure. > so you only get to just write a unit file and call it a day in the most trivial of cases. What "start/stop verification"? Give examples, please. Systemd…

> Now it can host telephony services, full-blown Docker orchestration systems, etc. What features that systemd provides that aren't provided by openrc, runit, monit, openrc's init system, etc, etc, etc that are needed by software like Asterisk, Docker/runc, Kubernetes, and similar? If what OpenWRT is running works fine to bring that up, then systemd doesn't have any relevant compelling features that aren't provided b…

This article explains the security features of systemd that make chroot superfluous: https://www.redhat.com/en/blog/mastering-systemd

And here's an example of how a network service can be confined in its own network namespace: https://www.cloudnull.io/2019/04/running-services-in-network... - with zero shell needed, btw.

Re: Why systemd is a problem for embedded Linux

#150

Post by kaba0 »

Aurornis wrote: post #110
jcalvinowens wrote: post #45

I couldn't disagree more: I've worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience. I won't pretend there aren't occasional weird problems... but there's always a solution, here's a recent example: https://github.com/systemd/systemd/issues/34683 Memory use is irrelevant to me: every embedded Linu…

> every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM. If I'm on a tiny machine where I care about 8MB RSS, I'm not running Linux, I'm running Zypher or FreeRTOS The gap between “over 1GB of RAM” and 8MB RSS contains the vast majority of embedded Linux devices. I, too, enjoy when the RAM budget is over 1GB. The majority of cost constrained products don’t allow that, though…

> It increases boot times

Is it really the case? On desktops it is significantly faster than all the other alternatives. Of course if you do know your hardware there is no need for discovering stuff and the like, but I don't know. Would be interested in real-life experiences because to me systemd's boot time was always way faster than supposedly simpler alternatives.

Post reply on HN