Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

131–140 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#131

Post by zh3 »

greatgib wrote: post #83
0x_rs wrote: post #80

journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that ha…

That was the task for years for syslog services that dealt with it without issue.

And still does. I generally replace journald with rsyslogd on systemd setups.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#132

Post by quotemstr »

pengaru wrote: post #127
quotemstr wrote: post #125

Earlier quoted context omitted.

> it's only used in a workaround when clearing FS_NOCOW_FL fails. Clearing FS_NOCOW_FL doesn't work on btrfs for non-empty files. So what do you think journald is doing when it notices that it can't clear the flag?

When did this become a discussion limited to journald on btrfs? and that seems like something btrfs should fix at some point

So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point.

> When did this become a discussion limited to journald on btrfs?

btrfs is in the HN thread title.

> and that seems like something btrfs should fix at some point

Amazing. The Linux kernel should change to work around journald's inflexibility?

What someone should fix at some point is journald's strange IO patterns and hard-coded "helpful" attribute changes. I'd rather it just rename the file and let me do any defrag/compression/flag-setting I want than do anything with chattr behind my back in ways I can't even configure.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#133

Post by xorcist »

e2le wrote: post #58
barrkel wrote: post #8

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…

I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.

If you want to store logs in a database, just use standard rsyslog. It has supported database backends pretty much since its inception at the dawn of the century. No need to reinvent anything.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#134

Post by kasabali »

magicalhippo wrote: post #94
barrkel wrote: post #8

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…

Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald. Why hasn't this been done if it's that terrible?

Because it's a big fat lie

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#135

Post by micw »

pseudalopex wrote: post #126
micw wrote: post #121

What does the (currently latest, https://github.com/systemd/systemd/issues/40262#issuecomment... ) comment mean? Who are the "large folio people"?

https://news.ycombinator.com/item?id=44113020

So is this a regression or a bug in systemd triggered by this change?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#136

Post by ptman »

amluto wrote: post #89
otterley wrote: post #75

Earlier quoted context omitted.

Is that true even with PRAGMA journal_mode = WAL? https://www.sqlite.org/wal.html

That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this. With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this. journald-style logs really w…

Clickhouse seems to be a popular logstore these days. And is column-oriented.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#137

Post by TylerE »

giov4 wrote: post #128
greatgib wrote: post #83

Earlier quoted context omitted.

That was the task for years for syslog services that dealt with it without issue.

I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful. all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them. syslog experience? very efficient also on heavy load production instances,…

WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#138

Post by pudgywalsh »

rasz wrote: post #117
breakingcups wrote: post #27

Earlier quoted context omitted.

But why, though? I have never seen a performance hit from it that would warrant that.

Dont like SSD wear for no reason. I wont look at those logs anyway on my personal gaming pc so its all useless.

Your issue is greatly exaggerated.

Enterprise users increase the logging and I've never heard of premature SSD failure due to this. The event log is capped in size (adjustable). It's nominally Your games continually dumping GBs of data into local cache on the other hand...

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#139

Post by actionfromafar »

TylerE wrote: post #137
giov4 wrote: post #128

Earlier quoted context omitted.

I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful. all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them. syslog experience? very efficient also on heavy load production instances,…

WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.

https://www.devuan.org/

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#140

Post by actionfromafar »

otterley wrote: post #37
quotemstr wrote: post #30

Earlier quoted context omitted.

SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.

Neither DuckDB nor LevelDB existed when journald was created. Not to say it couldn't be done today, but just some historical context.

LevelDB was released in 2011, so it existed but was very new.
Post reply on HN