41–50 of 114 posts
My view is pragmatic: if you are managing the server and I am a user, then use systemd if you want to. If it’s my responsibility then I will use a non-systemd distribution or if I have a choice, FreeBSD.
FreeBSD is many great things, but ... a pragmatic choice?
I've been a fan of systemd even if just for how much simpler creating services is. I just need to write a simple config file instead of a complex init script.
+1 - there was a misconception that initscripts were easy or simple - if you wanted _correct, bug free_ initscript you were probably looking at around 100 lines of shell script.
Now it has gone a bit overboard. Some of the stuff like the dns resolver or the nspawn capability seem a bit over the top, but overall, it has massively improved all Linux distributions it is used in.
Never again will I worry about trash buggy init scripts not actually stopping a service due to a stale pid file. Now it puts the service into a control group and can kill all things in the control group even if the service is bad code.
Does anyone actually use journald? The last time I tried(2 years ago?) it didn't even work with any log management software (like cloudwatch for example). You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
I built a small Ruby wrapper around the C API. Then I used that to slurp all the logs, periodically writing the current log ID to disk. Those logs went out onto a pubsub queue, where they were ingested into both BigQuery for long-term storage / querying, and into our alerting pipeline for real-time detection.
Thanks to journald, all the logs were structured and we were able to keep a bunch of trusted metadata like timestamp, PID, UID, the binary responsible, etc. (basically anything with an underscore prefix) separate from the log message all the way to BigQuery. No parsing, and you get free isolation of the trusted bits of metadata never intermingling with user-controlled attributes.
Compared to trying to durably tail a bunch of syslog files, or having a few SPOF syslog servers that everyone forwarded to, or implementing syslog plugins, this was basically the Promised Land for us. I think we went from idea to execution in maybe a month or two (I say “we” but really this was “me”) and rolled it out as a local daemon to the entire fleet of thousands. It has received—I think—one patch release in its six+ year lifetime, and still sits there quietly collecting everything to be shipped off-host.
The only issue we’ve ever really ran into that I never figured out is a handful of times per year (across a fleet of thousands) the journald database corrupted and you couldn’t resume collecting from the saved message ID. But we were also on an absolutely ancient version of RHEL, and I suspect anything newer probably fixed that bug. We just caught the error and restarted from an earlier timestamp. We built the whole thing around at-least-once delivery so having duplicates enter the pipeline didn’t really matter.
Damn, honestly at this point I’m wishing I’d pushed to open source it.
Ironically, actually, I did write a syslog server that also forwarded into this pipeline since we had network hardware we couldn’t install custom services onto but you could point them at syslog. I also wrote this in Ruby, using the new (at the time) Fibers (“real” concurrency) feature. The main thread fired up four background threads for listening (UDP, UDP/DTLS, TCP, TCP/TLS), and each of those would hand off clients to a dedicated per-connection worker thread for message parsing. Once parsed they went onto one more background thread for collecting and sending to PubSub. Even in Ruby it could handle gazillions of messages without breaking a sweat. Fun times.
Since I’m rambling, we also made cool use of zstd’s pre-trained dictionary feature. Log messages are small and very uniform so they were perfect for it. By pre-sharing a dictionary optimized for our specific data with both ends of the pubsub queue, we got something like 90%–95% compression rates. Given the many terabytes of logs we were schlepping from our datacenters to GCP, this was a pretty nice bit of savings.
Does anyone actually use journald? The last time I tried(2 years ago?) it didn't even work with any log management software (like cloudwatch for example). You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
One convenience of journald is that it exposes a single place to plug in log collection for observability tooling opentelemetry-collector, promtail, and so on have native plugins for it, which makes aggregation easier to setup Most tools have "tail this plaintext file" as well, but if it's all flowing to journald, setting up log collection ends up being that much simpler
I don't hate journald because it's not plaintext, I hate it because it's worse than plaintext. Somehow journald manages to provide a database which is 40x slower to query than running grep on a compressed text file. I'm all in favour of storing logs in an indexed structured format but journald ain't it.
The only use case where the binary format might make sense is if you ship journald logs to another central journald instance. That's just very much an edge case.
I don't hate journald because it's not plaintext, I hate it because it's worse than plaintext. Somehow journald manages to provide a database which is 40x slower to query than running grep on a compressed text file. I'm all in favour of storing logs in an indexed structured format but journald ain't it.
I now turn off journald and use rotated pain text log files. It's more efficient in all metrics.
No, it's terrible. I think what they mean by "success" here is that it functions, and is easier to use and more sturdy than init scripts. But any number of things would have been better than those. Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product. They basically mediate all Linux access to…
> Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product. Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you m…
And that quote proves 100% what is wrong with Linux these days.
In the 90s, if someone posted in USENET they want init to do X, at least 2 different solutions would appear with in a week, probably many more.
That happened to me when I went from Coherent OS to Slackware in the early 90s. I asked if there was a way to have virtual consoles on my VGA monitor and my B&W Hercules monitor (386sx), or to have at least 1 VC on the Hercules. Coherent 486 supported that, Linux did not. A post appeared in a couple of days with a patch.
Now, between the Linux Foundation and the Companies that owns it, that would never happen. People who have gotten into Linux in the last 20 years cannot imagine how responsive the developers were to user requests back then.
Seems these days we are dealing with some corporate behemoth drowning in red tape.
I am not a sysadmin -- so I keep forgetting the command to see logs cause I only need it once every 3 months. So, systemd change is a pain for me.
Better to spend a minute searching for a command in a file than to waste an hour recreating it.
I was following where the author was coming from until they quoted this, and rather than addressing it they just slammed it, slammed the author, and moved on.
It's weird to say that ini files are not a domain specific language. And this kind of hand waving away complexity is the crux of systemd putting a bad taste in my mouth. The whole thing is a vast kingdom of nouns (cf Yegge). Forgoing expressing semantics with the config file syntax means that expressing those semantics requires creating even more nouns. Splaying the config across a bunch of tiny files forgoes juxaposition and makes for even more nouns. Owning the aspect of whether a given config file is enabled or not creates even more nouns. Needing tools to analyze the distilled config (eg dependency graph) necessitates even more nouns.
... and this was all dumped on a community that very much focuses on a smaller number of verbs - ie command line utilities. The main problem with SysV scripts is that they are bare scripts doing anything they want with those longstanding verbs. So yes, some abstraction was necessary, and that was inevitably going to seem a bit unnecessary-complexity. But systemd just took that dynamic and went overboard with it.
Sure, I can read "type=forking" and know what that means, but no, I cannot change it without looking up the documentation. Any interaction with systemd beyond start/stop/journalctl involves looking up the documentation, re-figuring-out its implicit logical model, copying the right magical incantation, and then after I've solved the problem promptly forgetting everything I figured out because the next time I touch systemd I'm guaranteed to need to know something different.
I don't dislike it enough to desperately want to move to a different init system (although if there were an easy alternative on NixOS I'd try), and I'm certainly not going to defend SysV or BSD init scripts. But it certainly feels like there is plenty of room to implement systemd's comprehensive functionality in a more user-friendly discoverable way, especially for the main type of advanced-but-casual user that interacts with init systems.
(Also as a fellow NixOS user I think it's easy to have a blindspot about systemd because its config files align with our config format. A more user-friendly concise config format would make NixOS have to do more work to splay it out for its own composability)