Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

101–110 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#101

Post by quotemstr »

0x_rs wrote: post #80

journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that ha…

> I'd consider any btrfs install that does not have nocow on...to be defective.

You're getting COW on the extents if you're snapshotting anyway.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#102

Post by ButlerianJihad »

pudgywalsh wrote: post #12

How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly? The first thing I do on a Linux system is install a proper syslog daemon.

https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdJourna...

"The systemd journal doesn't force you to not have plain text logs" -- Chris Siebenmann (2024-06-30)

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#103

Post by vachina »

d3Xt3r wrote: post #32

Okay, so how do I disable journald and switch to something else, without getting rid of systemd completely?

I just set the storage to volatile and the max log size to 10Mbytes and call it a day.

Logs are really only useful at the tail.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#104

Post by emmelaich »

simoncion wrote: post #62
otterley wrote: post #34

Earlier quoted context omitted.

syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour... See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG... It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on…

> syslog doesn't have nearly the functionality that journald + journalctl does. A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks. Also, in regards to your original comment: > This issue report…

"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#105

Post by quotemstr »

amluto wrote: post #85
quotemstr wrote: post #46

Earlier quoted context omitted.

Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.

The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice. SQLite gets the last one but…

SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers.

Your LSM compaction strategy is going to have to solve the same problem anyway, isn't it? DuckDB is an LSM compaction strategy of this form, already done.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#106

Post by otterley »

emmelaich wrote: post #104
simoncion wrote: post #62

Earlier quoted context omitted.

> syslog doesn't have nearly the functionality that journald + journalctl does. A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks. Also, in regards to your original comment: > This issue report…

"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.

Indeed. It also means "don't come to me with problems alone." Yes, one can come with a problem, but the exhortation is to come with possible solutions as well and seek guidance on which one is best.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#107

Post by [deleted] »

hedora wrote: post #73
zbentley wrote: post #57

Earlier quoted context omitted.

What do you mean? That has described the vast majority of Linux systems I have ever touched, professionally or personally. Even corporate environments with log aggregation tail system logs rather than having them directly shipped elsewhere. The rare exceptions to this are some embedded devices without much durable storage, or tightly regulated environments in which log data is considered radioactive.

I’ve met people that think the log should be remotely stored and not written locally, since it’ll be shipped to splunk or whatever anyway. Those people change their minds the first time a machine has intermittent network issues, and the logs needed to debug it are lost (or worse, the log buffer fills, then stdout fills, which backpressures the application, creating an outage while simultaneously eating the logs).

[deleted]

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#108

Post by lokar »

zbentley wrote: post #57
lokar wrote: post #26

For 99% of installs the basic assumption that local logging (with local reading) is the primary mode is just wrong.

What do you mean? That has described the vast majority of Linux systems I have ever touched, professionally or personally. Even corporate environments with log aggregation tail system logs rather than having them directly shipped elsewhere. The rare exceptions to this are some embedded devices without much durable storage, or tightly regulated environments in which log data is considered radioactive.

By count, most installs will be the large cloud providers

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#109

Post by bothers »

otterley wrote: post #34
deepsun wrote: post #31

Earlier quoted context omitted.

Well, they are comparing in comments with syslog, and it does better, as you asked. Syslog was there for 46 years.

syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour... See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG... It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on…

Nothing says Linux like "information is hosted on a site owned by a user-hostile and privacy-mulching corporation."

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#110

Post by bothers »

emmelaich wrote: post #104
simoncion wrote: post #62

Earlier quoted context omitted.

> syslog doesn't have nearly the functionality that journald + journalctl does. A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks. Also, in regards to your original comment: > This issue report…

"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.

And it's always been cope. Only the worst kind of person would rather not know about a problem if it's not already solved for them.
Post reply on HN