Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why systemd is a problem for embedded Linux

kevinboone.me

Post reply on HN
Print view View original

111–120 of 308 posts

Re: Why systemd is a problem for embedded Linux

#111

Post by Aurornis »

jcalvinowens wrote: post #107
0xbadcafebee wrote: post #93

Earlier quoted context omitted.

> every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM That is almost by definition not an embedded device. There's a reason we have vfork().

Well, you can gatekeep all you want, but it's increasingly practical and common to have what would have seemed like an absurd amount of RAM a decade ago on things like toasters.

You have been living in a strange world if you’ve been getting away with 1GB in the average consumer IoT device for the past 5 years.

That’s not typical at all. I’ve done a lot of work with production runs in the millions. There is no way we’d put that much RAM on a device unless it was absolutely, undeniably required for core functionality.

Re: Why systemd is a problem for embedded Linux

#113

Post by cyberax »

simoncion wrote: post #102
cyberax wrote: post #72

Earlier quoted context omitted.

systemd binary is around 1.5Mb, with the shared libsystemd it's around 2.5Mb. Both can be reduced by not including compressors (the infamous libz), audit libraries, SELinux, etc. The minimum binaries size that can be obtained by trivial removals is around 2Mb. OpenWRT should just stop being stubborn and integrate it. They dropped support for 4Mb devices loooooong ago, and recently bumped the minimum specs to 16Mb. >…

> OpenWRT should just stop being stubborn and integrate it. Why? If it fails to provide benefits commensurate to the effort required to integrate it, then there's no reason to do the work. OpenWRT is for routers and access points... devices which usually have a fixed [hardware] configuration, and very little need for the absurdly-complex (and in my professional experience, often subtly buggy) event-driven systems tha…

OpenWRT is a mess. It started as a small distro for routers, but it stopped being that long ago.

Now it can host telephony services, full-blown Docker orchestration systems, etc. So you get all the pleasures of complex systems running on antiquated infrastructure.

> so you only get to just write a unit file and call it a day in the most trivial of cases.

What "start/stop verification"? Give examples, please. Systemd natively supports device dependencies, network status, other services, etc.

It's a very rare case when you _need_ shell in systemd.

Re: Why systemd is a problem for embedded Linux

#114

Post by somerandomqaguy »

0xbadcafebee wrote: post #93
jcalvinowens wrote: post #45

I couldn't disagree more: I've worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience. I won't pretend there aren't occasional weird problems... but there's always a solution, here's a recent example: https://github.com/systemd/systemd/issues/34683 Memory use is irrelevant to me: every embedded Linu…

> every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM That is almost by definition not an embedded device. There's a reason we have vfork().

Huh?

Been a while since I was in the digital signage space but a lot of the equipment runs of the shelf RK3288 plugged into commercial displays. 2GB of RAM was pretty common. IIRC's though LG's WebOS TV's have minimum 2GB of RAM in the digital signage space directly built into the units themselves. I believe Samsung Tizen based units has similar RAM.

My router has 1GB of RAM in it. But even my cheapest routers have 128 to 256 MB of RAM. The Cisco 9300 Catalyst switches have about 8 GB of RAM, but switches with beefy amounts of RAM are getting pretty common now, even if somewhat pricey.

Yeah there's massive swathes of embedded space that's tiny. But the higher end stuff isn't exactly out of reach anymore. The RK3288's IIRC ran about $20 a unit at the time before I left the industry.

Re: Why systemd is a problem for embedded Linux

#115

Post by cyberax »

jalgos_eminator wrote: post #108
cyberax wrote: post #72

Earlier quoted context omitted.

systemd binary is around 1.5Mb, with the shared libsystemd it's around 2.5Mb. Both can be reduced by not including compressors (the infamous libz), audit libraries, SELinux, etc. The minimum binaries size that can be obtained by trivial removals is around 2Mb. OpenWRT should just stop being stubborn and integrate it. They dropped support for 4Mb devices loooooong ago, and recently bumped the minimum specs to 16Mb. >…

OpenWRT is VERY sensitive to firmware size. I developed for a device with 16 MB flash that had both a normal (~11 MB) and recovery firmware (~3MB) on flash. The rest of flash was used for persistent storage, bootloaders, and calibration data. This is very common for older routers and APs which is the whole reason that OpenWRT exists.

Well, OpenWRT doesn't even officially support devices with less than 16Mb of flash anymore: https://openwrt.org/toh/views/toh_available_16128

> The rest of flash was used for persistent storage, bootloaders, and calibration data.

So you couldn't spare around 1Mb of compressed flash or 2Mb of uncompressed flash for systemd?

> This is very common for older routers and APs which is the whole reason that OpenWRT exists.

That hasn't been the case in quite some while.

And it won't matter anymore, there is no price difference between 32Mb and 64Mb NOR or NAND flash. It's literally the same cost.

Re: Why systemd is a problem for embedded Linux

#116

Post by jcalvinowens »

Aurornis wrote: post #111
jcalvinowens wrote: post #107

Earlier quoted context omitted.

Well, you can gatekeep all you want, but it's increasingly practical and common to have what would have seemed like an absurd amount of RAM a decade ago on things like toasters.

You have been living in a strange world if you’ve been getting away with 1GB in the average consumer IoT device for the past 5 years. That’s not typical at all. I’ve done a lot of work with production runs in the millions. There is no way we’d put that much RAM on a device unless it was absolutely, undeniably required for core functionality.

I've asked in the past, and been told that a even a 2x-3x difference in the amount of RAM made such a negligible difference in cost it was decided to go with the larger amount. I frankly have a hard time understanding how that can be true... but I can't really imagine why they wouldn't be honest with me about it.

Re: Why systemd is a problem for embedded Linux

#117

Post by throwaway2037 »

simoncion wrote: post #97
hactually wrote: post #58

> I wish I knew the solution to this problem Ooh ooh, I know! It's to improve systemd for the use-cases outlined and build a really great open source init/process manager.

The systemd maintainers are pretty particular about things that aren't a "supported use case", so even if you bother to do the work, you stand a solid chance of having your work be wasted. In my professional experience, "unsupported use case" effectively means "I don't want to work on that". [0] Also, at $DAYJOB, we run into mysterious systemd failures and misbehaviors at least once a year (usually for things that sh…

    > we run into mysterious systemd failures and misbehaviors at least once a year
Can you share an example?

Re: Why systemd is a problem for embedded Linux

#118

Post by tolciho »

0xbadcafebee wrote: post #93
jcalvinowens wrote: post #45

I couldn't disagree more: I've worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience. I won't pretend there aren't occasional weird problems... but there's always a solution, here's a recent example: https://github.com/systemd/systemd/issues/34683 Memory use is irrelevant to me: every embedded Linu…

> every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM That is almost by definition not an embedded device. There's a reason we have vfork().

Had vfork, unless you're on some not-BSD where vfork has remained relevant the last three decades?

DESCRIPTION vfork() was originally used to create new processes without fully copying the address space of the old process, which is horrendously inefficient in a paged environment. It was useful when the purpose of fork(2) would have been to create a new system context for an execve(2). Since fork(2) is now efficient, even in the above case, the need for vfork() has diminished. vfork() differs from fork(2) in that the parent is suspended until the child makes a call to execve(2) or an exit (either by a call to _exit(2) or abnormally). ... HISTORY The vfork() function call appeared in 3.0BSD with the additional semantics that the child process ran in the memory of the parent until it called execve(2) or exited. That sharing of memory was removed in 4.4BSD,

Re: Why systemd is a problem for embedded Linux

#119

Post by theamk »

m463 wrote: post #83

I think what he's running into is "the unix philosophy". It is basically about small tools, limited in scope, that solve a problem surgically and comprehensively. They work together by mixing and matching to solve the problem. systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing. I don't know, it is sort of like busybox, repla…

I think the problem is that small tools are missing that "comprehensively" part.. trying something as simple as getting full system startup log (something very useful on headless system!) - it was basically impossible until systemd came. Only gentoo tried to do this for a few years, but then that functionality broke.

Same goes for service startup - why does start-stop-daemon discards stdout/stderr instead of logging it? This would be trivial to fix, but somehow this was not available until upstart (and then systemd) did it.

Re: Why systemd is a problem for embedded Linux

#120

Post by amatecha »

strken wrote: post #57
rjzzleep wrote: post #29

Earlier quoted context omitted.

Random example. They randomly broke suspect-then-hibernate then the community manager gaslit people into trying to make them believe they don’t actually need the feature in the way it’s used and then silently acknowledged that their fix is broken after all. This is a common pattern by the way. But easily the most irritating thing for me is the simple issue that you used to be able to just go to /var/log/lastlog to se…

The linked GitHub issue is one of the most aggravating exchanges I've ever read: https://github.com/systemd/systemd/issues/25269 . I don't understand how you can read someone telling you that they want to suspend and then hibernate after a set duration, but A) not understand why this would be desirable, B) not understand that this is compatible with also hibernating at low battery, and C) not understand your own lack…

Yeah, that's pretty shitty behaviour, invalidating all the good-faith feedback and going so far as to call it "trolling"?? Ridiculous. There's no excuse to be so hostile when handling bug reports.
Post reply on HN