Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding Complexity with Systemd

mgdm.net

Post reply on HN
Print view View original

271–280 of 357 posts

Re: Avoiding Complexity with Systemd

#271

Post by geofft »

dec0dedab0de wrote: post #239
geofft wrote: post #104

Earlier quoted context omitted.

I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip (Not to ment…

Say the computer writing the log was destroyed, you were able to recover the storage, but the only computer available did not have journalctl. Or if you're writing to network storage and would like to analyze the logs from your haiku box. Text is not perfect, but it's the one thing that is always available. Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption we…

> the only computer available did not have journalctl

This is exactly what I don't understand. This is a world where no other computer exists? I have bigger things to worry about (even if you scope the problem down to to "no other computer with journalctl installed exists on my network").

> Or if you're writing to network storage and would like to analyze the logs from your haiku box.

I don't have any Haiku boxes, but I do have Windows boxes. At my day job, where I'm a Linux sysadmin for a large finance company, my workstation is Windows, and I don't have admin on it. So even with conventional UNIX logs, a much more common case is - as I mentioned - that I'd want to read them from a computer that doesn't have gzip installed.

But we're fine with gzip logs, because the way I'd actually do this is to get a Linux computer running.

(Also, keep in mind that the filesystem itself is a binary format. If you're really worried about reading logs from Haiku, you wouldn't put /var/log on NFS because that sounds like a terrible idea, you'd log to a FAT filesystem. But nobody actually does that. Everyone's logs are on ext4 or XFS or btrfs or whatever, and nobody says those formats are a bad idea.)

Re: Avoiding Complexity with Systemd

#272

Post by pessimizer »

nonameiguess wrote: post #181
josephcsible wrote: post #119

Earlier quoted context omitted.

What does this have to do with job policies? Systemd was forced down our throats by Red Hat adding hard dependencies on it to other software under their control, e.g., GNOME. Other distros then adopted systemd under duress, since many of the packages that Red Hat made depend on it are important to the Linux ecosystem, and the other distros didn't have the resources to fork them all.

The point is that the replacements being talked about, systemd-resolved for dns, systemd-timesyncd for ntp, and systemd-networkd for whatever else you would prefer to use for network config, are not a mandatory part of systemd. You can use systemd without using these other components and it will work perfectly well with whatever other services you want to use for dns, ntp, and ip networking. These other services are…

You have to be an expert on systemd to figure out how not to use systemd.

Re: Avoiding Complexity with Systemd

#273

Post by ghoward »

bkor wrote: post #259
josephcsible wrote: post #236

Earlier quoted context omitted.

> That you can only say things such as "forced down our throats" and "duress" says enough. Not capable to actually hold a discussion, let's be emotional and without any actual facts. Does this sound any better? Red Hat used their influence over GNOME (and other programs) to add a hard dependency on systemd to it. This forced other distros to either switch to systemd or drop support for GNOME. I suspect that had Red H…

Nope, it's still entirely incorrect. Again, the release team misjudged things. I was part of the GNOME release team at that time. We were actually warned about it, then misjudged it ("it'll be fine"). Further, it wasn't even a hard dependency. You're really not understanding components and APIs. > This forced other distros to either switch to systemd or drop support for GNOME. No, again entirely incorrect. GNOME runs…

Not GP.

The fact that it took _work_ to get GNOME to run without systemd is a bad thing in my book.

Also, to claim that systemd was selected on merit without anything backing up the claim of "merit" is disingenuous; plenty of worse solutions end up winning all the time.

Re: Avoiding Complexity with Systemd

#274

Post by gwillen »

geofft wrote: post #271
dec0dedab0de wrote: post #239

Earlier quoted context omitted.

Say the computer writing the log was destroyed, you were able to recover the storage, but the only computer available did not have journalctl. Or if you're writing to network storage and would like to analyze the logs from your haiku box. Text is not perfect, but it's the one thing that is always available. Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption we…

> the only computer available did not have journalctl This is exactly what I don't understand. This is a world where no other computer exists? I have bigger things to worry about (even if you scope the problem down to to "no other computer with journalctl installed exists on my network"). > Or if you're writing to network storage and would like to analyze the logs from your haiku box. I don't have any Haiku boxes, bu…

Note that not every journald/journalctl is created equal. It's easy to end up with logs written on one computer that journalctl on another computer can't read, even if the latter is a newer version, depending on which settings each one was compiled with (which is mostly a distro question.)

Re: Avoiding Complexity with Systemd

#275

Post by WesolyKubeczek »

mcspiff wrote: post #263
gmmeyer wrote: post #253

Earlier quoted context omitted.

I have never encountered the need for this and I honestly doubt you have either. Systemd fails mysteriously far less often than poorly written init files I have seen and it is honestly not that hard to debug almost every failure I've ever encountered. And I have worked on some very large scale systems with systemd. A systemd file I helped write has run many millions of times over without issue. I honestly find this c…

I’m a big fan of systemd, but we’ve definitely run into actual bugs in it. Especially the early RHEL7 days (7.0-7.3). Some were fixed upstream, some we’ve worked around, etc.

In the beginning, the road was quite bumpy, and my irk was that people's use cases were being dismissed in a very cavalier fashion: like, if I didn't think of a workflow, or a use case, or a piece of software that doesn't play nice which you cannot change, it's not valid and let it burn. Oh, and whoever points these things out, or ridicules me for this attitude, is a troll or a hater.

In 10 years, though, reading recent bug reports, I can see that the project's leads have grown out of it, largely.

Case in point, they had recently put a bug into a 24x release that made a lot of machines, including mine, unbootable, but the fix was just as quick.

Re: Avoiding Complexity with Systemd

#276

Post by WesolyKubeczek »

lanstin wrote: post #226
spinax wrote: post #200

Earlier quoted context omitted.

> added a log trimmer config for squid I feel your pain - I'm curious, was it a drop-in /etc/logrotate.d/ config which was sending a HUP? I don't run squid but Google'd up that it will take a USR1 to rotate logs which should not close the HTTP connections (allegedly HUP closes them). Perhaps the AL2 folks chose the wrong signal? https://wiki.squid-cache.org/SquidFaq/InstallingSquid#squid_...

But systemd itself was setting the squid service into stop then start. I don’t think by that time systemd knew the reason was to do with logs.

I'm wondering if this new config made squid exceed any limits, causing it to hard stop. I'd expect the log trimmer to have/cause a bug in squid itself.

Re: Avoiding Complexity with Systemd

#277

Post by gwillen »

geofft wrote: post #104
einpoklum wrote: post #100

Earlier quoted context omitted.

1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. 2. Some of the systemd logging is binary, so good luck with that if there's a problem. 3. Have you tried non-systemd init systems other than sysvinit? 4. Yes, it is convenient when everything below your development is cenrtalized by a single entity. It can easily provide a consistently useful underpinning. But…

I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip (Not to ment…

Binary logging would be fine if it Just Worked, but my experience is that journald/journalctl does not just work, and when it breaks I don't have logs.

For example, if the system clock is not monotonic, my text logs are still written sequentially in order, and it's easy to figure out what happens. It is the stated view of the journald maintainers that, if your clock is not monotonic, you get to keep both pieces. https://github.com/systemd/systemd/issues/662 (The specific issue described there sounds complicated, but the fundamental problem behind it, which I have hit multiple times, is "if your system can't maintain a monotonic clock at all times, including early boot, then your logs will be mangled.")

Similarly, a text log line is complete the moment it's written, even if the process that wrote it goes away immediately afterwards. Journald, by contrast, performs asynchronous metadata lookups for _each_ log line at some later time, which it apparently does not cache. This means that, when a process dies suddenly, the final messages will not appear in "journalctl -u", because they failed to get tagged with metadata. (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.)

So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.

Re: Avoiding Complexity with Systemd

#278

Post by dane-pgp »

kaba0 wrote: post #150
dane-pgp wrote: post #111

Earlier quoted context omitted.

I believe that Red Hat employees were some of the main contributors to Gnome, which quickly made systemd a hard requirement, forcing distros which used it as the default desktop to make systemd the default init system.

How about the two democratic votes in favor of debian made in a system that is more democratic than real world voting? Or that Arch independently made the switch?

You're missing the initial Debian vote, which was made in the Technical Committee, and which came down to a tie.

If you want to claim that process was democratic, you have to believe that the members of the committee perfectly represent the opinions of all Debian developers (to say nothing of Debian users), and therefore have to excuse the fact that the vote had to be settled by giving one of those people, Bdale Garbee (HP's CTO of Linux), effectively two votes.

It was only three days after the tech committee's decision that Mark Shuttleworth announced that Ubuntu would fall in line by abandoning Upstart[0], and not until many months later that the first General Resolution was put forward to try overturning the committee.[1] Of course by that time everyone was tired of the arguments and it would have soured relations with Canonical to force Ubuntu back away from systemd, so the GR was doomed from the start.

[0] https://www.admin-magazine.com/News/Ubuntu-Abandons-Upstart

[1] https://www.debian.org/vote/2014/vote_003

Re: Avoiding Complexity with Systemd

#279

Post by InvertedRhodium »

WesolyKubeczek wrote: post #275
mcspiff wrote: post #263

Earlier quoted context omitted.

I’m a big fan of systemd, but we’ve definitely run into actual bugs in it. Especially the early RHEL7 days (7.0-7.3). Some were fixed upstream, some we’ve worked around, etc.

In the beginning, the road was quite bumpy, and my irk was that people's use cases were being dismissed in a very cavalier fashion: like, if I didn't think of a workflow, or a use case, or a piece of software that doesn't play nice which you cannot change, it's not valid and let it burn. Oh, and whoever points these things out, or ridicules me for this attitude, is a troll or a hater. In 10 years, though, reading rec…

There is still no way to filter application output at collection time, which can functionally make journald useless if you have an application that is too chatty, and will likely never be given that (AFAICT) there is hefty ideological opposition from the systems developers¹.

¹https://github.com/systemd/systemd/issues/6432

Re: Avoiding Complexity with Systemd

#280

Post by zlynx »

dec0dedab0de wrote: post #239
geofft wrote: post #104

Earlier quoted context omitted.

I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip (Not to ment…

Say the computer writing the log was destroyed, you were able to recover the storage, but the only computer available did not have journalctl. Or if you're writing to network storage and would like to analyze the logs from your haiku box. Text is not perfect, but it's the one thing that is always available. Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption we…

The journalctl binary format seems to handle corruption pretty well. That was a design criteria.

Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT. It is encoded in 7-bit ASCII with records delimited by 0x0a bytes.

Corruption tends to be missing data, and so the reader has to jump ahead to find the next synchronization byte, aka 0x0a. This also leads to log parsers producing complete trash as they try to parse a line that has a new timestamp right in the middle of it.

Or there's a 4K block containing some text and then padded to the end with 0x00 bytes. And then the log continues adding more after reboot. Again, that's fixed by ignoring data until the next non-zero byte and/or 0x0a byte. This problem makes it really obvious that text logs are binary files.

See the format definition at https://www.freedesktop.org/wiki/Software/systemd/journal-fi...

And here, this isn't perfect but if you had to hack out the text with no journalctl available you could try this:

grep -a -z 'SYSLOG_TIMESTAMP=\|MESSAGE=' /var/log/journal/69d27b356a94476da859461d3a3bc6fd/system@4fd7dfdde574402786d1a1ab2575f8fb-0000000001fc01f1-0005c59a802abcff.journal | sed -e 's/SYSLOG_TIMESTAMP=\|MESSAGE=/\n&/g'

Post reply on HN