What does the (currently latest, https://github.com/systemd/systemd/issues/40262#issuecomment...) comment mean? Who are the "large folio people"?
Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
121–130 of 239 posts
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#122journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that ha…
That was the task for years for syslog services that dealt with it without issue.
rsyslog is an incredible piece of software. Every time I'm looking for something to do with logs, opening the docs or googling finds the feature for me and myriads alternatives. I know it still exists and use it heavily on any system I'm in charge of, but there's some regret at having a dual system with journalctl...
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#123I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
I wish Qubes Domain-0 was a customized Gentoo with OpenRC. Fedora with systemd was a poor choice to base off. Nobody should have let Poettering have the influence he was given over userland, systemd is an almost irrevocable mistake.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#124Earlier quoted context omitted.
Write-Ahead Log for... logs? WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread. I believe journald log rotation is basically: close file - open a new one. How is it not completely different?
journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates. https://github.com/systemd/systemd/blob/8f4cd7de43d1e6e94687... WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working. So, yes, WAL for logs, because LSM is the de…
> journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails.
Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming unused space.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#125Earlier quoted context omitted.
journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates. https://github.com/systemd/systemd/blob/8f4cd7de43d1e6e94687... WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working. So, yes, WAL for logs, because LSM is the de…
> journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates. You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails. Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming u…
> it's only used in a workaround when clearing FS_NOCOW_FL fails.
Clearing FS_NOCOW_FL doesn't work on btrfs for non-empty files. So what do you think journald is doing when it notices that it can't clear the flag?
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#126What does the (currently latest, https://github.com/systemd/systemd/issues/40262#issuecomment... ) comment mean? Who are the "large folio people"?
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#127Earlier quoted context omitted.
> journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates. You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails. Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming u…
> it's only used in a workaround when clearing FS_NOCOW_FL fails. Clearing FS_NOCOW_FL doesn't work on btrfs for non-empty files. So what do you think journald is doing when it notices that it can't clear the flag?
When did this become a discussion limited to journald on btrfs?
and that seems like something btrfs should fix at some point
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#128journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that ha…
That was the task for years for syslog services that dealt with it without issue.
I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful.
all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them.
syslog experience? very efficient also on heavy load production instances, never lost a log, pipe grep and jq and you have the info you need.
so what I experienced is that a default linux install was shipping a rock solid logging system by default, reliable and usable and everybody knew what was where and you will find it. now i just have fancy stuff, units etc and lost all of that.
no I dont need to tune config parameters on a default install to have working basic logging tnx.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#129Earlier quoted context omitted.
> syslog doesn't have nearly the functionality that journald + journalctl does. A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks. Also, in regards to your original comment: > This issue report…
"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.
Yes, a lot of awful practices have always been a thing.
> e.g. hook in a senior engineer that you know is intimate with the system.
unless, of course, you don't know said engineer because you don't even work in the same company, you're a just a user seeing a problem in an app you use
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#130Earlier quoted context omitted.
Well, they are comparing in comments with syslog, and it does better, as you asked. Syslog was there for 46 years.
syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour... See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG... It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on…
You forgot to link to the tests that prove it