41–50 of 239 posts
Earlier quoted context omitted.
Why not just have a SQLite file and call it a day? Also, why mmaped file?
SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.
Systemd should just use DuckDB. It's perfect for this job. "But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?" Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data. BTW: no, you don't need O(…
You can read them with DuckDB, but you don't end up with O(log n) writes -- which is, to speak plain English, batshit fucking insane for a system logger.
What those cursed writes buys you is O(log n) reads, but there's just no scenario that is necessary. If you have literally any time or subsystem constraints, parquet's predicate pushdowns means you get plenty fast access even with a full scan.
Earlier quoted context omitted.
SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately. It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years). If you're interested, here are my fin…
Earlier quoted context omitted.
Why not just have a SQLite file and call it a day? Also, why mmaped file?
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic. There was mailing list discussion at the time journald was conceived though, you can find it if you look. https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
Earlier quoted context omitted.
Thank you for your work. ISTM the workload is naturally LSM-shaped. > If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written. Exactly. So either make the format append-only or make it append-mostly with occasional write…
Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.
I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately. It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years). If you're interested, here are my fin…
Have you considered using a different fstype like XFS for this? btrfs is good for homedirs, but I wouldn't necessarily use it for other filesystems (/usr, /var, etc.)
How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly? The first thing I do on a Linux system is install a proper syslog daemon.
I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…
Thank you for your service!
Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luckily was only possible because it were kernel messages (dmesg), but for a while I had to disae persistent kernel logging which is dat from ideal.
I get that for certain core parts simplicity is more important than features. But journald is just too basic to enable persistent storage but I also don't want to switch it off.