Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

141–150 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#141
post #52

Earlier quoted context omitted.

The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...

FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.

... Sounds like a signature on a patch that triggers an epic Linus rant on LKML[1]

[1] Happened few times, I think RedHat as a whole even got banned from sending changes for a short while

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#142
post #71

Earlier quoted context omitted.

Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.

Or, you could write a plain text file. Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice. If you really ca…

Honestly, I would go append-only blocks that contain binary/compressed format with synchronizing marks so worst case you get some nulls but every reader can synchronize where they are in the stream without blocking anyone.

Might take more space on disk than theoretical best of journald storage format with its absurd hashtables, but it fulfills the job of system log better and more complex format should be done in log aggregation layer.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#143

This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.

>This issue report feels like it ought to be accompanied by a fix. systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such. These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals…

In this case, it's also a very ego-driven project.

Honestly, the few times I went into systemd source (to deal with how they didn't document some critical information without which I couldn't ensure coexistence of other software, software needed for functionality systemd didn't expose), I found it a total mess - combined with very loud and explicit ways the decision of the "stewards" were defended by the team, I would be frankly wary of trying to contribute anything non-trivial.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#144
post #13

I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…

I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right?

On the other hand, if I was ever to advise anyone on how to do I/O when they are working with an (unknown) filesystem... It's really hard. And I'd probably default to saying "do as few tricks as possible" because filesystems today are very elaborate, with a lot of optimizations that are very difficult to predict from user-space. It's quite possible that someone trying to outsmart a filesystem will end up harming themselves in the process.

Doing as few tricks as possible would allow the administrator to configure the filesystem independently of the program writing to it to match the nature of the workload instead of locking the program into a specific pattern of operation that might be impossible to rectify with administrative tools. Not an ideal situation by any means: storage-heavy user-space applications s.a. databases usually do the opposite: they try to optimize for the specific filesystem, its version and quirks... but it takes a lot of effort, obviously.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#145

Earlier quoted context omitted.

> syslog doesn't have nearly the functionality that journald + journalctl does. A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks. Also, in regards to your original comment: > This issue report…

"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.

I would argue the difference is that upstream is well known for being defensive about their ideas and pushing back.

On LKML you might get cursed out, but if your fix is solid fix, it has high chances of getting through. Regardless of how true it would be in reality, the atmosphere created by upstream is that I do not expect the same with journald unless you convince redhat management

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#146

Something must have happened along the way, because this was not the original design intent of the database (emphasis mine): """ The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the n…

I would argue the described format is exactly the origin of the problem.

It tries to optimize on disk footprint by deduplication and resulting in way more complex file format with many possible footguns leading to things like write amplification while also making it less robust for the actual use cases of a persistent log.

In a way, it's using a file format more useful for aggregation layer, except it doesn't do that well either, compromising immediate needs at local level.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#147
post #4

Earlier quoted context omitted.

Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately. It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years). If you're interested, here are my fin…

> This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes Have you considered using a different fstype like XFS for this? btrfs is good for homedirs, but I wouldn't necessarily use it for other filesystems (/usr, /var, etc.)

I don't have experience with xfs or zfs, and I don't have a free stand for experiments right now unfortunately.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#148

years ago, I set Storage=volatile on almost all the journalD configurations I have. This largely solved this kind of problem.

Good for your personal devices, very not good for servers where you need to have any sort of accountability and security trail.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#149

Earlier quoted context omitted.

Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald. Why hasn't this been done if it's that terrible?

You can trivially configure it to forward logs to another service to manage them.

that's not the point, why ship most common linux distros with an unreliable logging solution by default?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#150
post #2

https://github.com/systemd/systemd/issues/40262#issuecomment...

Thanks for that comment. Out of curiosity how did you come up with that setup? For me most of that test suite sounds alien

If it weren't mmaped files, I would use strace/gdb, or even fuse proxy file system.

But these are mmaped, I don't know any easy debugging or monitoring solution besides writing kprobes/systemtap hooks.

How would you debug it?

Post reply on HN