My hunch having looked at the journald code as an amateur is that this write amplification is coming from scattering, with a few possible sources:
1. Writes try to compress away duplicate metadata at the application layer, which causes them to issue scattered writes when new metadata shows up.
2. Indexing is also surprisingly log-line/application-layer aware, such that index writes might also be scattering.
3. The indexes themselves seem like they could benefit from an append-mostly write model with periodic compaction rather than a mutate-in-place model.
4. I was surprised that the journal’s “WAL” doesn’t seem to be a major concern of a lot of the code. For a database, supporting reads “through” the WAL with periodic application back to the data files (“checkpoints” in RDBMS) seems like something I’d expect to see more of here. But I don’t really have deep understanding of the code, so I may be missing that it’s doing that already.
The choice of mmap instead of regular file writes here isn’t, as others have proposed, a design flaw. I think that makes sense given what journald is (a database) and how significant its durability concerns are. And it looks like the code does spend a lot of time trying to be careful about which blocks/pages are dirtied. But this is a famously hard-to-get-write (ha!) area so perhaps defects are present at that layer.
The systemd developers are talented in their area; I am not a systemd hater. However, “talented at low-level OS design” is not the same as “talented at building a database from scratch”, and I think that shows here.
I strongly feel like this system could be a wrapper around SQLite, which is definitely something that could be integrated everywhere journald is used (license-wise and compatibility-wise). I’m puzzled as to why that wasn’t chosen as an approach: a SQLite vfs implementation that handled compression and online rotation seems like it would have resulted in a design that’s both more interoperable and less prone to flaws like this one.
I also think that a per-log-emitter setting that doesn’t eagerly persist to disk (wait for page cache flush) would be very useful to have available—perhaps even as a default—for user-level/init6 level logs that are OK with a potential for data loss on kernel panic.