Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

211–220 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#211
post #137

Earlier quoted context omitted.

WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.

Gentoo also has a systemd-free option (and even if it didn't, you could make one because Gentoo is basically LinuxFromScratch)

WHAT R UR CFLAGS??!?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#212
post #193

Earlier quoted context omitted.

There is a certain consistency to IBM-dominated projects that suggests that there are checks and balances and that they are working as intended.

This design predates the IBM acquisition by seven years.

And Poettering worked at Microsoft until recently.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#213

Earlier quoted context omitted.

Good for your personal devices, very not good for servers where you need to have any sort of accountability and security trail.

That’s why you ship the logs off host to a central collector.

Nice in theory in practice you always retain them locally as well, just in case the network connection goes down.

If network egress fails and logs are pushed in real time over a connection with no local backing, you face an ugly tradeoff: either drop log data silently (loss of visibility during the very network partition you need to debug) or apply backpressure to services (potentially hanging applications when logging buffers saturate).

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#215

Earlier quoted context omitted.

That’s why you ship the logs off host to a central collector.

Nice in theory in practice you always retain them locally as well, just in case the network connection goes down. If network egress fails and logs are pushed in real time over a connection with no local backing, you face an ugly tradeoff: either drop log data silently (loss of visibility during the very network partition you need to debug) or apply backpressure to services (potentially hanging applications when loggi…

Agreed. That's what the log agent's disk buffer is for. That can still be used even if the journal itself is on volatile storage.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#216

Earlier quoted context omitted.

facepalm Way to double down on the “needlessly picky and combative” angle, dude.

> Way to double down on the “needlessly picky and combative” angle, dude. Sit and consider the points of similarity between my refusal-shaped reply and the entire conversation we had prior to it and you might find enlightenment, in the style of those classic Zen tales. Perhaps an LLM-based tool might be able to assist you in this, or maybe it will be distracting and misleading. GLHF and all that.

> you might find enlightenment, in the style of those classic Zen tales

Doctor, heal thyself.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#217

Earlier quoted context omitted.

The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.

I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower than just pulling out ripgrep on regular files. We have some services at work that log to text files and some to journald. The log volume to file is >> the log volume to journald. Yet `rg query myservice.2026-08-01.log` seems to always wind up being faster and…

What do the metrics look like in practice? seems is a bit too handwavy. I get that impressions matter, but data is actionable.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#218

Earlier quoted context omitted.

Sure, a concrete proposal first would be a good idea. That said, would you look a gift horse in the mouth?

It doesn't matter how free a turd sandwich is.

There's no proposal that we can evaluate to determine whether it's a turd sandwich or not.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#219

Earlier quoted context omitted.

systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?

Probably because the overall impact is not as bad as extremely vocal people on GitHub and HN would have you believe, and more people like systemd than dislike it.

[dead]

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#220

Earlier quoted context omitted.

Say more? Sounds like a good story

My caveman understanding is that mmap writes are bad for transactional accesses because you have little to no control over sync. The OS can decide to commit changes to disk anytime, in any order which is the opposite of what you want for anything ressembling a database.

The problems are much more than just sync.

A long time ago I had this over-optimistic idea: x86 hardware (and probably most other hardware) has these cool hardware-managed dirty page bits. So you would write to a mapped page, not even take a page fault, and the hardware would record that it's dirty. Later on the kernel would notice and flush. Excellent performance.

Hahaha. It's much much much more complex. For various reasons (maybe good, maybe bad -- see below), Linux barely uses the real hardware dirty bit. Instead, when you map a file as shared-writable, at first it might not really be mapped at all. If you read it, it gets faulted in and becomes readable. When you first try to write to it, a page fault is generated, and, on non-FRED x86, the page fault itself is very slow. The kernel will do things, including calling into the FS and updating atime [0], to make the page logically writable. It updates the page tables so that the CPU knows it's writable, and it sets the dirty bit right then (after all, this is a bit faster than letting the CPU set it immediately thereafter when you retry the faulting write).

Okay, now it's writable. Writes are essentially free until the kernel decides to write the data back to the disk. The kernel will mark the page non-writable (because it wants to get notified the next time you try to write to it) and flush the TLB (which is extremely expensive, especially on x86 systems that aren't the latest AMD CPUs). And it will write the page back, more or less as if you had used normal syscalls to write it.

There's more fun, though. Some filesystems and/or backing stores need "stable pages" -- they need the page cache pages that are being written to not be modified while being written back. btrfs, for example, wants to checksum the data and then write the data and the checksum out consistently, and if something changes the data while it's being DMAed, then this can't happen. So special locks might be taken to delay future writes to the page until writeback is done, and that includes blocking the "make writable" page fault handler. Oops, there goes performance.

Could the kernel do better? Probably. Will it? Unlikely in the near future. I've contemplated a special mechanism to map a "fast write" window onto a file that would be permanently writable and use the hardware dirty bit to tell the kernel when to transfer the data out. Even if anyone ever implemented this, it would be a very specialized thing, it would incur polling overhead, and it would be utterly silly to use it for something like syslog.

Just use pwrite or io_uring unless you have actual evidence that mmap is better.

mmap read is a different story, of course.

[0] I think that updating atime at make-writable time instead of at writeback time is both non-performant and semantically incorrect. I've never convinced the maintainers well enough, though.

Post reply on HN