Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

231–239 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#231

Post by soneil »

marginalia_nu wrote: post #222
bombela wrote: post #221

Earlier quoted context omitted.

I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020. $ time journalctl > /tmp/all.log real 1m11.364s user 0m52.299s sys 0m6.540s $ time wc -l /tmp/all.log 3659597 /tmp/all.log real 0m0.152s user 0m0.056s sys 0m0.096s $ time journalctl | grep sshd | wc -l 12944 real 0m53.…

I'm no fan of journald, but I have some methodological issues with this test. The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either. Running this between calls should flush the changes to disk and then drop the page c…

I tried similar. Each command looped 5 times, ignore the best and worst times, and average the mean three.

    journalctl >/tmp/all.log  0m59.763s
    journalctl | grep -c sshd 0m57.767s  (otherwise >all is the only example encumbered by write speeds)
    wc -l /tmp/all.log        0m0.336s
    grep -c sshd /tmp/all.log 0m1.776s

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#232

Post by otterley »

soneil wrote: post #231
marginalia_nu wrote: post #222

Earlier quoted context omitted.

I'm no fan of journald, but I have some methodological issues with this test. The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either. Running this between calls should flush the changes to disk and then drop the page c…

I tried similar. Each command looped 5 times, ignore the best and worst times, and average the mean three. journalctl >/tmp/all.log 0m59.763s journalctl | grep -c sshd 0m57.767s (otherwise >all is the only example encumbered by write speeds) wc -l /tmp/all.log 0m0.336s grep -c sshd /tmp/all.log 0m1.776s

You didn't use the query feature of journalctl, so you basically had it do an entire table conversion. Of course that's not going to show its performance value.

This is what you should be running for a proper comparison:

1. `echo 3 | sudo tee /proc/sys/vm/drop_caches`

2. `time journalctl -u ssh.service >/dev/null`

3. `journalctl >/tmp/all.log`

4. `echo 3 | sudo tee /proc/sys/vm/drop_caches`

5. `time grep -q sshd /tmp/all.log`

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#233

Post by emmelaich »

simoncion wrote: post #205
eviks wrote: post #129

Earlier quoted context omitted.

Yes, a lot of awful practices have always been a thing. > e.g. hook in a senior engineer that you know is intimate with the system. unless, of course, you don't know said engineer because you don't even work in the same company, you're a just a user seeing a problem in an app you use

> unless, of course, you don't know said engineer because you don't even work in the same company... Or they work in a different part of the fairly-large company that you both work for. I guess emmelaich either missed the part of my commentary where I talked about handling inter-organization communication, and/or has never worked at a company where it's simply impossible to know everyone who could reasonably be relev…

So put out feelers. Ask everyone if they know anyone....

At least triage it with your best effort.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#234

Post by jck86 »

mzajc wrote: post #87
jck86 wrote: post #50

The cherry on the cake is that you practically cannot filter journald. The only option is limiting by severity (e.g. errors and higher) or switch to non persistent journald storage and forward to rsyslog and filter there. Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luc…

If you have systemd>=253 you can make use of LogFilterPatterns[0] (in .service files), but it's really unpredictable, cumbersome to work with, and does not work with user services or non-service log sources. [0]: https://www.freedesktop.org/software/systemd/man/latest/syst...

Thanks for this, was not aware this was added. Though better would be to have a global option for this, and a tunable whether to filter this completely or only for persistent storage.

For the last few days I have been monitoring journald with iotop and found in my case storage use was not excessive at the moment. And there are rate limit options, but I'd really like there to be system-wide filtering options.

For servers I rarely see journald persistence enabled while it is actually very valuable for debugging crashes and other issues. Way easier than regular log files. Though also more fragile and more difficult, so improvements are very welcome.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#235

Post by fghj117 »

crabbone wrote: post #144
pengaru wrote: post #13

I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…

I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right? On…

[dead]

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#236

Post by valyala »

amluto wrote: post #89
otterley wrote: post #75

Earlier quoted context omitted.

Is that true even with PRAGMA journal_mode = WAL? https://www.sqlite.org/wal.html

That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this. With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this. journald-style logs really w…

VictoriaLogs could be a good alternative: it stores logs in column-oriented compressed format, so they occupy small amounts of disk space, and it consists of a single small executable without external dependencies, which stores the logs into a single directory on a local filesystem.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#237

Post by Hundredth0006 »

greatgib wrote: post #83
0x_rs wrote: post #80

journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that ha…

That was the task for years for syslog services that dealt with it without issue.

syslog wasn't dealing with it because all of this output was written in .xsession-errors in user's homedir (which was not rotated btw)

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#238

Post by NewJazz »

ptman wrote: post #136
amluto wrote: post #89

Earlier quoted context omitted.

That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this. With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this. journald-style logs really w…

Clickhouse seems to be a popular logstore these days. And is column-oriented.

Isn't it at least somewhat heavy weight though?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#239

Post by VGHN7XDuOXPAzol »

otterley wrote: post #217
VGHN7XDuOXPAzol wrote: post #192

Earlier quoted context omitted.

I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower than just pulling out ripgrep on regular files. We have some services at work that log to text files and some to journald. The log volume to file is >> the log volume to journald. Yet `rg query myservice.2026-08-01.log` seems to always wind up being faster and…

What do the metrics look like in practice? seems is a bit too handwavy. I get that impressions matter, but data is actionable.

Sure, I'll try my best to put some concrete numbers on it:

On a file of ~1M lines (350 MB), ripgrep returns a query matching 22k lines in 1.9s.

(Separately: We have gzip log rotation for old logs. A log with 1.9M lines (106 MB gzipped) on disk can be searched from cold in 1.3s by ripgrep, whilst still being ergonomic. Maybe a different compressor is better?)

On a service unit filter over ~218k lines, `journalctl -u service --since -g query` returns 4.8k matching lines in 4.127s (and this is after having warmed the cache by returning the unfiltered query cold, in 11.9s. I don't have root to clear the disk cache

I should have ran the filtered query first but ah well, since the difference is already so large it doesn't matter that the filtered journalctl query gets an unfair advantage)

Comparing input rates:

    ripgrep (plain): 576k line/sec
    ripgrep (gz): 1.4M line/sec
    journalctl (-u, --since, -g): 53k line/sec
I don't have a great deal of understanding of journald's internals, so perhaps there is some variable here that is unreasonably unfavourable to journald.

Transcript

    user@machine:~/log$ time rg query aservice.log | wc -l
    22628

    real    0m1.859s
    user    0m0.109s
    sys     0m0.909s
    user@machine:~/log$ wc -l aservice.log
    1071606 aservice4.log


    user@machine:~$ time journalctl --since '2026-08-16' -u myservice@1.service | wc -l
    218661

    real    0m11.914s
    user    0m10.729s
    sys     0m0.798s

    user@machine:~$ time journalctl --since '2026-08-16' -u myservice@1.service -g query | wc -l
    4804

    real    0m4.127s
    user    0m3.727s
    sys     0m0.186s
Probably more painful on the day to day is how `journalctl -fu myservice` seems to stall for on the order of 5 to 10s, sometimes. I can't reproduce at the moment and maybe it only happens on some machines, but if you were interested in 'real-world anecdata' it's something to maybe note.
Post reply on HN