[flagged]
Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
151–160 of 239 posts
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#152Earlier quoted context omitted.
Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component
Sure, a concrete proposal first would be a good idea. That said, would you look a gift horse in the mouth?
It doesn't matter how free a turd sandwich is.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#153I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…
I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right? On…
Libtorrent 2.0 switched exclusively to mmaped read/writes for torrent downloads, which resulted in various performance and especially memory consumption issues on ALL platforms.
For some reason Windows handled increased memory consumption the least gracefully.
Many people continued to use v1.2 which use regular files.
V2.1 ended up using pread/pwrite nowz it's fine now.
The issue continued for 3 years more or less.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#154Earlier quoted context omitted.
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic. There was mailing list discussion at the time journald was conceived though, you can find it if you look. https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment...
It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#155[dead]
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#156journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that ha…
I'm using a certain object storage implementation post minio enshitification. The software itself is great don't get me wrong but I've noticed their logs are basically unreadable. Its metrics and traces in json data meant to be rendered on a dashboard instead of being read by humans. It's also extremely verbose even at an INFO level.
Maybe just get these on an open telemetry endpoint instead? I also don't get why people send by default json logs to journald as it's clearly meant to be a replacement to syslog which is already a good standard.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#157I completely agree with one of the comments from there:
> But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better.
It really does not make any sense to use memory-mapped files when writing logs.
Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files.
Only when reading logs, to search for problems, accessing them as read-only memory-mapped files is OK.
Actually not only for logs, but almost always, read-write memory-mapped files are either inefficient or too complex to use (i.e. to avoid problems you must carefully use msync and/or madvise, which eliminates the simplicity that makes memory-mapped files preferable to using pread/pwrite). It is better to use memory-mapped files only for read-only accesses, using the appropriate option flags in open and mmap.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#158Okay, so how do I disable journald and switch to something else, without getting rid of systemd completely?
I just set the storage to volatile and the max log size to 10Mbytes and call it a day. Logs are really only useful at the tail.
Having the previous boot can get you out of a hole if you're dealing with dodgy drivers.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#159Earlier quoted context omitted.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately. It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years). If you're interested, here are my fin…
> This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes Have you considered using a different fstype like XFS for this? btrfs is good for homedirs, but I wouldn't necessarily use it for other filesystems (/usr, /var, etc.)
You lose rollbacks, superior syncing, redundancy and data integrity assurance and add complexity for what deceased ssd wear which is hardly an actual problem.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#160I completely agree with one of the comments from there: > But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better. It really does not make any sense to use memory-mapped files when writing logs. Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files. Only when rea…
great and clear summary thank you!
I would laso add that if my design decisions or development actions lead to an issue affecting multiple linux distro defaults I would feel responsible and rush for a solid fix instead of this https://github.com/systemd/systemd/issues/15292#issuecomment...