journald has never been of great quality. It somehow manages to be visibly slower than grepping gzipped text logs.
Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
11–20 of 239 posts
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#12How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#13I'm probably the main person responsible for making journald usable at all.
But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.
Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.
My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.
Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#14I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…
journald uses hash tables, I think it update it on every new log line, although I didn't debug it in depth yet.
https://github.com/systemd/systemd/blob/199f75205b9c0625bf56...
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#15[flagged]
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#16journald has never been of great quality. It somehow manages to be visibly slower than grepping gzipped text logs.
it has bad scaling properties esp. if you have many journal files
the last time I contributed to journald upstream was to fix a degenerate behavior with many journal files: https://github.com/systemd/systemd/commit/176f73272e6e3116ca...
that makes a dramatic difference for those hitting this case, but it only gets things from nearly unusable to slow-as-usual.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#17I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…
Why not just have a SQLite file and call it a day?
Also, why mmaped file?
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#18journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…
I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#19I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…
Why not just have a SQLite file and call it a day? Also, why mmaped file?
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic.
There was mailing list discussion at the time journald was conceived though, you can find it if you look.
https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes
#20Oh how I love totally predictable poetterings reply to previous bug report that got closed because "measuring it wrong" and "this is not a support forum".