Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

221–230 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#221

Post by bombela »

otterley wrote: post #217
VGHN7XDuOXPAzol wrote: post #192

Earlier quoted context omitted.

I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower than just pulling out ripgrep on regular files. We have some services at work that log to text files and some to journald. The log volume to file is >> the log volume to journald. Yet `rg query myservice.2026-08-01.log` seems to always wind up being faster and…

What do the metrics look like in practice? seems is a bit too handwavy. I get that impressions matter, but data is actionable.

I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020.

$ time journalctl > /tmp/all.log

real 1m11.364s user 0m52.299s sys 0m6.540s

$ time wc -l /tmp/all.log 3659597 /tmp/all.log

real 0m0.152s user 0m0.056s sys 0m0.096s

$ time journalctl | grep sshd | wc -l

12944

real 0m53.973s user 0m49.535s sys 0m5.210s

$ time grep sshd /tmp/all.log | wc -l 12944

real 0m0.429s user 0m0.332s sys 0m0.100s

https://github.com/systemd/systemd/issues/2460#issuecomment-...

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#222

Post by marginalia_nu »

bombela wrote: post #221
otterley wrote: post #217

Earlier quoted context omitted.

What do the metrics look like in practice? seems is a bit too handwavy. I get that impressions matter, but data is actionable.

I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020. $ time journalctl > /tmp/all.log real 1m11.364s user 0m52.299s sys 0m6.540s $ time wc -l /tmp/all.log 3659597 /tmp/all.log real 0m0.152s user 0m0.056s sys 0m0.096s $ time journalctl | grep sshd | wc -l 12944 real 0m53.…

I'm no fan of journald, but I have some methodological issues with this test.

The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either.

Running this between calls should flush the changes to disk and then drop the page cache, making for a fairer test.

$ sudo sync

$ echo 3 | sudo tee /proc/sys/vm/drop_caches

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#223

Post by quotemstr »

pengaru wrote: post #209
quotemstr wrote: post #132

Earlier quoted context omitted.

So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point. > When did this become a discussion limited to journald on btrfs? btrfs is in the HN thread title. > and that seems like something btrfs should fix at some point…

> btrfs is in the HN thread title. as is ext4

So write amplification on btrfs doesn't matter?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#224

Post by quotemstr »

amluto wrote: post #186
quotemstr wrote: post #118

Earlier quoted context omitted.

How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file? Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doi…

On the disk the live update part would be either a circular buffer (and readers would need to double-check the start/end marks after reading) or just literal append-only streams. In the latter case, reading would be barely more complex than tail -f.

Sure, but you have to take pains to make sure that journalctl -f doesn't skip events or print some twice. It can be made to work, for sure, but it just seems easier to print logs via a daemon instead, especially because if you go through a deamon, you turn the disk format from an interface into an implementation detail.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#225

Post by sam_lowry_ »

NewJazz wrote: post #212
otterley wrote: post #196

Earlier quoted context omitted.

This design predates the IBM acquisition by seven years.

And Poettering worked at Microsoft until recently.

And the went on to make age checks mandatory on Linux, earning money in the process.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#226

Post by sam_lowry_ »

hdgvhicv wrote: post #166
TylerE wrote: post #137

Earlier quoted context omitted.

WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.

The problem with systemd is the scale. Some of it is fine, some of it is actually quite good. But other areas are just replacing existing systems with things worse for the majority of traditional users. Logging, time and dns come to mind.

systemd-boot ia actually one of the rare components that are pleasant to work with.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#227

Post by [deleted] »

brohee wrote: post #167
giov4 wrote: post #154

Earlier quoted context omitted.

I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment... It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.

Ah, the Ulrich Drepper school of dealing with reported issues. Time for esystemd ;)

[deleted]

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#228

Post by sam_lowry_ »

d3Xt3r wrote: post #32

Okay, so how do I disable journald and switch to something else, without getting rid of systemd completely?

Put the following in etc/systemd/journald.conf.d/discard.conf

    [Journal]
    Storage=none
    ForwardToSyslog=no
    ForwardToKMsg=no
    ForwardToConsole=no
    ForwardToWall=no
Then restart with

    systemctl restart systemd-journald

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#229

Post by rasz »

pudgywalsh wrote: post #138
rasz wrote: post #117

Earlier quoted context omitted.

Dont like SSD wear for no reason. I wont look at those logs anyway on my personal gaming pc so its all useless.

Your issue is greatly exaggerated. Enterprise users increase the logging and I've never heard of premature SSD failure due to this. The event log is capped in size (adjustable). It's nominally Your games continually dumping GBs of data into local cache on the other hand...

Size cap doesnt mean much when its a constant stream of small writes.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#230

Post by amluto »

quotemstr wrote: post #224
amluto wrote: post #186

Earlier quoted context omitted.

On the disk the live update part would be either a circular buffer (and readers would need to double-check the start/end marks after reading) or just literal append-only streams. In the latter case, reading would be barely more complex than tail -f.

Sure, but you have to take pains to make sure that journalctl -f doesn't skip events or print some twice. It can be made to work, for sure, but it just seems easier to print logs via a daemon instead, especially because if you go through a deamon, you turn the disk format from an interface into an implementation detail.

I want my logs readable on a completely read-only disk. Sure, I can start a daemon, but that’s mildly annoying.

Also, the syslog daemon should be extremely reliable, and throwing giant table scans at it makes this more complex.

Post reply on HN