Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

91–100 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#91

Post by edoceo »

throw-the-towel wrote: post #48
pudgywalsh wrote: post #12

How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly? The first thing I do on a Linux system is install a proper syslog daemon.

Which log daemon do you use?

syslog-ng FTW! Been using it since like 2003. Great router tools in the conf, network support, crazy regex rules for when you're trying to tune your syslog collector system after a few beers. Even has this awesome (footgun?) feature that lets me pipe matching lines to other tools I wrote (also beer influenced).

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#93

Post by Walf »

e2le wrote: post #58
barrkel wrote: post #8

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…

I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.

Text or text-like (e.g. text content with simple control char delimiters for metadata) would be far superior than the slow-down from Sqlite's safety mechanisms. Optimising logs for read, at the expense of write, is a bad pattern to me.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#94

Post by magicalhippo »

barrkel wrote: post #8

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…

Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald.

Why hasn't this been done if it's that terrible?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#95

Post by TingPing »

magicalhippo wrote: post #94
barrkel wrote: post #8

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…

Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald. Why hasn't this been done if it's that terrible?

You can trivially configure it to forward logs to another service to manage them.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#96

Post by simoncion »

otterley wrote: post #86
simoncion wrote: post #84

Earlier quoted context omitted.

You: [0] Nobody's talking about an obligation here. Also you: [1] If you think you can do better than journald's existing format, propose a new one with tests to prove it. The fact that you're personally powerless to enforce an obligation doesn't change the fact that you're talking about creating an obligation. > I'm not entirely sure this is a bug so much as an inefficiency report. Performance bugs absolutely are bu…

Well, I didn't intend to suggest an obligation, more of a best practice or challenge. Please put more faith in my intentions over whatever interpretation of my words you want to make. Per our Guidelines: > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

> Please respond to the strongest plausible interpretation... Assume good faith.

That's what I did. So, right back at you.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#97

Post by otterley »

simoncion wrote: post #96
otterley wrote: post #86

Earlier quoted context omitted.

Well, I didn't intend to suggest an obligation, more of a best practice or challenge. Please put more faith in my intentions over whatever interpretation of my words you want to make. Per our Guidelines: > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

> Please respond to the strongest plausible interpretation... Assume good faith. That's what I did. So, right back at you.

> That's what I did

Please explain, because it's not coming across that way. It's coming across as needlessly picky and combative, especially after I told you what I meant (or, at least, didn't mean) and you continued to argue with me.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#98

Post by ahartmetz »

Walf wrote: post #93
e2le wrote: post #58

Earlier quoted context omitted.

I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.

Text or text-like (e.g. text content with simple control char delimiters for metadata) would be far superior than the slow-down from Sqlite's safety mechanisms. Optimising logs for read, at the expense of write, is a bad pattern to me.

Read optimized? That is funny because reading logs from journald is dog slow compared to, you know, log files.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#99

Post by CrimsonCape »

marginalia_nu wrote: post #38
d3Xt3r wrote: post #32

Okay, so how do I disable journald and switch to something else, without getting rid of systemd completely?

https://www.freedesktop.org/software/systemd/man/latest/syst... See StandardOutput= and StandardError=.

Ok, so I should set these to null? I saw elsewhere someone set journald storage to volatile. Which of these approaches is better?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#100

Post by [deleted] »

amluto wrote: post #85
quotemstr wrote: post #46

Earlier quoted context omitted.

Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.

The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice. SQLite gets the last one but…

[deleted]
Post reply on HN