Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

111–120 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#111

Post by tomjakubowski »

e2le wrote: post #58
barrkel wrote: post #8

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…

I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.

There's an open source project which is a syslog daemon that stores logs in DuckDB. You can configure journald to forward logs to it.

https://github.com/phare/sloggo

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#112

Post by amluto »

quotemstr wrote: post #105
amluto wrote: post #85

Earlier quoted context omitted.

The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice. SQLite gets the last one but…

SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers. Your LSM com…

I don’t think there’s any hard work here. Other than the append-only part, all files would be either immutable (the Parquet parts) or maybe mutated by wholesale atomic replacement of the inode (the catalog, although the directory itself, via its contained filenames) could maybe do that. Readers might have to retry sometimes, but readers would neither have boy expect any write privileges.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#113

Post by dchest »

quotemstr wrote: post #43
ElectricalUnion wrote: post #41

Earlier quoted context omitted.

No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.

Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.) We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.

Write-Ahead Log for... logs?

WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread.

I believe journald log rotation is basically: close file - open a new one. How is it not completely different?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#114

Post by dmitrygr »

cloudie78 wrote: post #17
pengaru wrote: post #13

I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…

Why not just have a SQLite file and call it a day? Also, why mmaped file?

Because Poettering didn’t invent SQLite.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#115

Post by dchest »

otterley wrote: post #75
rcxdude wrote: post #70

Earlier quoted context omitted.

sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get…

Is that true even with PRAGMA journal_mode = WAL? https://www.sqlite.org/wal.html

Yes, WAL by definition means it writes the data at least twice.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#116

Post by rasz »

sidewndr46 wrote: post #68
rasz wrote: post #22

Earlier quoted context omitted.

One of the first things I do on win10 is disable most of excess logging.

How do you do that?

Painfully and slowly clicking one item at a time in computer management/event viewer/windows logs/applications and services :|

Im sure this can be automated, but I want to see what Im disabling instead of going bulk all.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#117

Post by rasz »

breakingcups wrote: post #27
rasz wrote: post #22

Earlier quoted context omitted.

One of the first things I do on win10 is disable most of excess logging.

But why, though? I have never seen a performance hit from it that would warrant that.

Dont like SSD wear for no reason. I wont look at those logs anyway on my personal gaming pc so its all useless.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#118

Post by quotemstr »

amluto wrote: post #112
quotemstr wrote: post #105

Earlier quoted context omitted.

SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers. Your LSM com…

I don’t think there’s any hard work here. Other than the append-only part, all files would be either immutable (the Parquet parts) or maybe mutated by wholesale atomic replacement of the inode (the catalog, although the directory itself, via its contained filenames) could maybe do that. Readers might have to retry sometimes, but readers would neither have boy expect any write privileges.

How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file?

Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doing any less efficient than what your custom Parquet thing would be doing?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#119

Post by quotemstr »

dchest wrote: post #113
quotemstr wrote: post #43

Earlier quoted context omitted.

Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.) We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.

Write-Ahead Log for... logs? WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread. I believe journald log rotation is basically: close file - open a new one. How is it not completely different?

journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.

https://github.com/systemd/systemd/blob/8f4cd7de43d1e6e94687...

WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working.

So, yes, WAL for logs, because LSM is the design everyone converges on and a WAL is LSM. Better to use the LSM already implemented and debugged in a database than write some random new one in terms of Parquet that's going to have to do the same stuff in the end anyway, just with novel bugs and no tool support.

(And look, I don't give a damn what "DB" people say, a WAL writing back to a DB IS log... structured... merge under any fucking sensible definition of what LSM means.)

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#120

Post by micw »

0x_rs wrote: post #80

journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that ha…

systemd-journald also has rate limits that you can configure ;-)
Post reply on HN