Live data from Hacker News

Why systemd is a problem for embedded Linux

kevinboone.me

271–280 of 308 posts

Re: Why systemd is a problem for embedded Linux

#271

Earlier quoted context omitted.

That's entirely the point. In the days of user devices with 32MB of RAM, embedded devices were expected to make do with 32KB. Now we have desktops with 32GB and the embedded devices have to make do with 32MB. But you don't get to use GB of RAM now just because embedded devices might have that in some years time, and unless something is done to address it, the increase in hardware over time doesn't get you within the…

I recently looked up 2x48GB RAM kits and they are around 300€ and more for the overclockable ones. That is 3€ per GB and this is in the more expensive segment in the market since anyone who isn't overclocking their RAM is fine using four slots.

The end of that chart is in 2020 and in the interim the DRAM makers have been thumped for price fixing again, causing a non-trivial short-term reduction in price. But if this is the "real" price then it has declined from ~$10/GB in 2012 to, let's say, $1/GB now, a factor of 10 in twelve years. By way of comparison, between 1995 and 2005 (ten years, not twelve) it fell by a factor of something like 700.

You can say the free lunch is still there, but it's gone from a buffet to a stick of celery.

Re: Why systemd is a problem for embedded Linux

#272
post #261

Earlier quoted context omitted.

> now I have to avoid that and minimise tinkering/hacking. I think this right here hits at the crux of the issue. There are people who like systemd because it's integration-tested with itself and its own defaults, so if you never change those defaults you don't have many problems. Then there are people who don't like systemd because if you do have to change any of its defaults, it often doesn't go well. And, of cours…

> And, of course, the latter behavior as a box users are expected to live in is poisonous, because everyone is being conditioned to be passive and uniform. No, it's not, this assumes that the other camp are all idiots. Mainstream distros should be rock solid and boring. Linux never got anywhere with the standard distros because they're all so different. Tinkering is a different mindset (and has a different place) tha…

> Linux never got anywhere with the standard distros because they're all so different.

This is eliding the difference in what you want to be standardized.

Take system logging for example. You definitely want some standard interface to do it so all the different daemons and things can implement it once regardless of which system logger the distribution is using. But once it passes that data to the other program, it's under a different dominion of control which should operate as a black box with respect to the program generating the data and the rest of the system.

Because that's how you prevent ossification -- which is engineering. You want to make it so the other component can be improved or substituted. One system logger is designed to store the logs on a central server instead of the local machine, another supports modules so other people can easily write log-parsing scripts. And when you come up with a new idea for a third, you don't have to be on the systemd team to publish an independent implementation that other people can use as a drop-in replacement, instead of (not in addition to) the default one, without requiring it to be separately integrated with a dozen moving targets in the systemd repository.

What you're trying to do is to allow this:

> In the same vein, don't be creative with bolt sizes. Be creative with what those bolts allow you to achieve. Be creative at a higher level.

You want to standardize the bolt sizes, i.e. the interfaces between components. What you explicitly don't want is to replace bolts with having everything epoxied together.

But that's what happens when you e.g. integrate the logging system with the service manager, which is the sort of thing people are complaining about.

Re: Why systemd is a problem for embedded Linux

#273

Earlier quoted context omitted.

I'd bet you get more CVEs and actual remote exploit entry points by systemd forcing mdns, bonjour, upnp and other zero conf hacks just because that was the work the systemd team was doing in rh before.

remembered another one. They use the same code for VM and bare metal. Including the easy-login convenience hacks. So now anyone wanting physical access to your host, just have to intercept plain-text communication with the BIOS (after secureboot did its thing), and reply with a new root password when the OS request bios key 11 or something. evil maids are living the dream.

Is that with systemd logind? Why does logind interact with BIOS?

https://vincent.bernat.ch/en/blog/2021-startx-systemd

Re: Why systemd is a problem for embedded Linux

#274
post #260

Earlier quoted context omitted.

The real question is would anything else also be able to do 99.9999% of the job with less code and better semantics. People seem to think it is.

Poettering actually took his idea to completion. A lot of people claim the same, but the proof is in the real life pudding. Don't forget that geeks idealize perfection and especially minimalist perfection.

> Poettering actually took his idea to completion.

One of them. The others didn't work out so well. That's a curiosity in and of itself.

> A lot of people claim the same, but the proof is in the real life pudding.

Those claims are true. Those systems exist. People use them. Every day. I've got tons of business riding on them. Systemd's defaults are simply a non starter for any real production environment.

> Don't forget that geeks idealize perfection and especially minimalist perfection.

They prefer a system which leaves the options open. Systemd removes them in the name of simplistic cohesion. It's all based on the flawed idea that linux will ever be a popular desktop operating system if you just make it work like windows does.

It misses the point on what computing is entirely, but hey, as long as it's "complete" implementation of this flawed vision I guess that's good enough for some people. And yet we're still no where near windows.

Re: Why systemd is a problem for embedded Linux

#275
post #257

Earlier quoted context omitted.

I thought I saw in the last release a note that the next one would come with systemd. It could be I hallucinated it. Maybe I'm becoming a real LLM!

RELEASE_NOTES [1] contains no mention of systemd (or at least does not contain that string). README.TXT [2] also does not contain the string "systemd" ANNOUNCE.15.0 [3] also does not contain the string "systemd" If such a 'note' were included in the last release (15.0) it should have been in one of the above three files. So there is nothing in the release that appears to indicate a shift to systemd. [1] https://ftp.u…

Slackware will never implement systemd. Father would not allow it.

Re: Why systemd is a problem for embedded Linux

#276

Earlier quoted context omitted.

> So you couldn't spare around 1Mb of compressed flash or 2Mb of uncompressed flash for systemd? Absolutely not. > That hasn't been the case in quite some while These routers and APs still exist and are in use. Yes, they are 10+ years old, but they're still out there. One of the main goals of OpenWRT is to support legacy networking hardware that vendors abandoned.

> Absolutely not. So basically, you're dealing with devices that use obsolete hardware and are designed to be extra-brittle (no safety margin), with a built-in planned obsolescence (what if the new update requires just a bit more space for "calibration data"?). Got it. sysvinit indeed fits perfectly. Make the system extra unreliable, to provide a stimulus to buy a newer version. > One of the main goals of OpenWRT is…

OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices.

> And OpenWRT abandoned this goal.

They have not. Individual devices get too old to be supported, but there are plenty of 16 MB devices they still support. Its true that basically every device now has 32MB or more of flash, but there are still plenty out there that don't.

> with a built-in planned obsolescence (what if the new update requires just a bit more space for "calibration data"?). Got it.

The calibration data is from the factory to tune the wifi radios, it doesn't change since its specific to that individual device. I really hate to reply like this on HN, but you don't seem to know what you're talking about. The kinds of constraints you deal with in embedded networking are different than that in general computing. There's a reason systemd wasn't used, and there's a reason OpenWRT devs went through the trouble of developing their own init system. Just because you can't fathom those reasons doesn't mean they don't exist.

Re: Why systemd is a problem for embedded Linux

#277

Earlier quoted context omitted.

But desktops with those specs are If they can't afford those specs then you're approaching the group of people who can't afford a computer in the first place.

Yes, that's about 3 billion people. They use whatever they can afford. Usually that's whatever is very old, because new software doesn't run well except on very new hardware which is more expensive. I recently wanted an MP3 player for an art project. Local stores don't sell mp3 players anymore, they only sell smartphones. So I bought an MVNO smartphone for $40. When I charged it up and tried to use it, I thought mayb…

This is simply so misleading thats its nonsense.

So you bought a new phone for $40, and it was a POS?

My kids use my old iPhone 7, which is in the same price bracket and is nothing like that. Its fast enough for Roblox, Minecraft, and certainly fast enough for a web browser.

I have an old Dell USFF that I use for server purposes, but its a Skylake (so newer than what was the original conversation), with ssd and 16Gb, and that was Because thats what were talking about.

> But desktops with those specs are I just checked eBay. Yes they still are.

Re: Why systemd is a problem for embedded Linux

#278

Earlier quoted context omitted.

> Absolutely not. So basically, you're dealing with devices that use obsolete hardware and are designed to be extra-brittle (no safety margin), with a built-in planned obsolescence (what if the new update requires just a bit more space for "calibration data"?). Got it. sysvinit indeed fits perfectly. Make the system extra unreliable, to provide a stimulus to buy a newer version. > One of the main goals of OpenWRT is…

OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices. > And OpenWRT abandoned this goal. They have not. Individual devices get too old to be supported, but there are plenty of 16 MB devices they still support. Its true that basically every device now has 32MB or more of flash, but there are still plenty out…

> OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices.

procd is only slightly better than sysvinit, it has the same ideas, and only adds simple and incorrectly implemented event triggers. And yes, I spent days debugging issues that it caused.

E.g. a USB drive for the logs becomes slow and changes the startup order, so now openvpn starts before the interface acquires the IPv6 address. Boom, broken IPv6 routing.

> They have not. Individual devices get too old to be supported, but there are plenty of 16 MB devices they still support. Its true that basically every device now has 32MB or more of flash, but there are still plenty out there that don't.

And systemd can live just fine at 16MB. I ran it on RouterBoards with 32MB (the smallest ones available) in 2014. It has grown a bit since then, but it can be slimmed down (remove compressors, auditing, SELinux/AppArmor, etc.).

> The calibration data is from the factory to tune the wifi radios, it doesn't change since its specific to that individual device. I really hate to reply like this on HN, but you don't seem to know what you're talking about.

The entire firmware for WiFi chips that were available during 16MB era is around 500KB-1MB. So it must be some other mysterious "calibration" data.

So you're thinking about more and more corner cases, that require the stars to be just right for the size of systemd to matter.

Re: Why systemd is a problem for embedded Linux

#279

Earlier quoted context omitted.

Except those $2.50 for customer will turn into $2500000.00 for the hardware vendor. And they would rather keep the $2500000.00 and let it be someone else's problem.

If you're shipping 1 million units, don't use Linux. Done.

You are implying hardware vendor is the one who got to choose the OS, but that's often not how it works.

Re: Why systemd is a problem for embedded Linux

#280
post #247

Earlier quoted context omitted.

I switched to systemd when Arch switched and from the get go, it was massively easier to parallelise with systemd than with the old system and that was with an HDD. Systemd already parallelises by default so I don't know what insanely strange things you were doing but I fail to see how it could bring boot time form 11s to 1 minute. Also, it's very easy to get a list of every services enabled with systemctl (systemctl…

Running things in parallel isn't going to make the disk faster... with a HDD I'd think it is actually even more likely to make the disk slower.

We don’t have to talk in hypotheticals here. Booting time benchmarks from the time systemd was released are everywhere and showed shorter boot times. It was discussed ad nauseam at the time.
Post reply on HN