Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

51–60 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#51

Post by marginalia_nu »

doublepg23 wrote: post #9
ValdikSS wrote: post #4

Earlier quoted context omitted.

Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately. It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years). If you're interested, here are my fin…

The two I most often see in Ubuntu's dmesg are: audit - appears to be some sort of AppArmor logging? br[] - bridge interface docker uses consistently rebuilds itself? May be related to docker compose networking.

Yeah docker does a ton of network stuff when you start/stop containers, depending on your configuration. It's extra fun because it can drop existing connections when that happens.

Had a process quietly in a crash loop for a solid month on my workstation until I figured out what was causing my random network outages.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#52

Post by pengaru »

otterley wrote: post #45
pengaru wrote: post #19

Earlier quoted context omitted.

I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic. There was mailing list discussion at the time journald was conceived though, you can find it if you look. https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.

The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...

FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#53

Post by pengaru »

ValdikSS wrote: post #14
pengaru wrote: post #13

I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…

journald uses hash tables, I think it update it on every new log line, although I didn't debug it in depth yet. https://github.com/systemd/systemd/blob/199f75205b9c0625bf56...

The format is documented

https://github.com/systemd/systemd/blob/main/docs/JOURNAL_FI...

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#55

Post by orf »

marginalia_nu wrote: post #42
quotemstr wrote: post #25

Systemd should just use DuckDB. It's perfect for this job. "But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?" Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data. BTW: no, you don't need O(…

Parquet is probably an even better option. Columnar, compression, fast, succinct. All good things. You can read them with DuckDB, but you don't end up with O(log n) writes -- which is, to speak plain English, batshit fucking insane for a system logger. What those cursed writes buys you is O(log n) reads, but there's just no scenario that is necessary. If you have literally any time or subsystem constraints, parquet's…

No, not at all. Parquet is great for building static content incrementally, but it’s not great for this: the aim is durable writes (it’s a log system after all), but with parquet you need large row group batches. Worst case (low log volumes and a time-based flush) you’d end up with loads of tiny row groups.

You also need metadata in the file footer, so you can’t query it until the file is “done”. When is that?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#57

Post by zbentley »

lokar wrote: post #26

For 99% of installs the basic assumption that local logging (with local reading) is the primary mode is just wrong.

What do you mean? That has described the vast majority of Linux systems I have ever touched, professionally or personally. Even corporate environments with log aggregation tail system logs rather than having them directly shipped elsewhere. The rare exceptions to this are some embedded devices without much durable storage, or tightly regulated environments in which log data is considered radioactive.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#58

Post by e2le »

barrkel wrote: post #8

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…

I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#59

Post by simoncion »

otterley wrote: post #39

Something must have happened along the way, because this was not the original design intent of the database (emphasis mine): """ The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the n…

If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality.

Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped in with additional information. [0] My hunch is that the current set of people working on the Systemd Project are going to be supremely disinterested in fixing the problem... and might even be entirely unable to fix it. A project this large and sprawling that runs for this long without a solid commitment to quality doesn't tend to retain many very highly-skilled individuals.

[0] https://news.ycombinator.com/item?id=49291376>

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#60

Post by e2le »

quotemstr wrote: post #30
cloudie78 wrote: post #17

Earlier quoted context omitted.

Why not just have a SQLite file and call it a day? Also, why mmaped file?

SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.

Sqlite3 is present in the default installation of most Linux distributions. It has proven itself from years of battle testing in many different environments. To use DuckDB or LevelDB would probably require pulling in an additional dependency.
Post reply on HN