Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

181–190 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#181

Post by smartmic »

giov4 wrote: post #160
adrian_b wrote: post #157

I completely agree with one of the comments from there: > But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better. It really does not make any sense to use memory-mapped files when writing logs. Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files. Only when rea…

great and clear summary thank you! I would laso add that if my design decisions or development actions lead to an issue affecting multiple linux distro defaults I would feel responsible and rush for a solid fix instead of this https://github.com/systemd/systemd/issues/15292#issuecomment...

This is really astonishing. Are there no checks and balances in place for design decision in such a critical system component? What were the thoughts of all the major distros when they decided to go with systemd then?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#182

Post by zamadatix »

micw wrote: post #120
0x_rs wrote: post #80

journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that ha…

systemd-journald also has rate limits that you can configure ;-)

IIRC those are "dumb" rate limits though, aren't they? I.e. if 1000 of the same dumb message comes across from the service then you'll rate limit out the 3 useful messages occurring at the same time.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#183

Post by pineapplepizza6 »

otterley wrote: post #77
rcxdude wrote: post #76

Earlier quoted context omitted.

The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.

The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.

[dead]

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#184

Post by otterley »

pineapplepizza6 wrote: post #176
giov4 wrote: post #154

Earlier quoted context omitted.

I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment... It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.

systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?

Probably because the overall impact is not as bad as extremely vocal people on GitHub and HN would have you believe, and more people like systemd than dislike it.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#185

Post by p_l »

pineapplepizza6 wrote: post #177
p_l wrote: post #142

Earlier quoted context omitted.

Honestly, I would go append-only blocks that contain binary/compressed format with synchronizing marks so worst case you get some nulls but every reader can synchronize where they are in the stream without blocking anyone. Might take more space on disk than theoretical best of journald storage format with its absurd hashtables, but it fulfills the job of system log better and more complex format should be done in log…

You could even generate a bloom filter for every block, and when you have a full block of bloom filters, write them out to an index file.

My personal idle walking-with-dog kind of design was a linear binary record file with regular marks letting you resynchronize where you are (and stamp cryptographically) with minimal seeks, and separate indexing files with bloom filters and the like. If the indexes are corrupted or deleted, they can be reconstructed from the main log, main log is single-writer/multiple-readers with no locking in any form necessary, and easier to survive kernel/hw failure

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#186

Post by amluto »

quotemstr wrote: post #118
amluto wrote: post #112

Earlier quoted context omitted.

I don’t think there’s any hard work here. Other than the append-only part, all files would be either immutable (the Parquet parts) or maybe mutated by wholesale atomic replacement of the inode (the catalog, although the directory itself, via its contained filenames) could maybe do that. Readers might have to retry sometimes, but readers would neither have boy expect any write privileges.

How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file? Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doi…

On the disk the live update part would be either a circular buffer (and readers would need to double-check the start/end marks after reading) or just literal append-only streams. In the latter case, reading would be barely more complex than tail -f.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#187

Post by Throwaway123129 »

adrian_b wrote: post #157

I completely agree with one of the comments from there: > But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better. It really does not make any sense to use memory-mapped files when writing logs. Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files. Only when rea…

I wonder if something like https://github.com/open-telemetry/otel-arrow would be a better fit instead

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#188

Post by pineapplepizza6 »

smartmic wrote: post #181
giov4 wrote: post #160

Earlier quoted context omitted.

great and clear summary thank you! I would laso add that if my design decisions or development actions lead to an issue affecting multiple linux distro defaults I would feel responsible and rush for a solid fix instead of this https://github.com/systemd/systemd/issues/15292#issuecomment...

This is really astonishing. Are there no checks and balances in place for design decision in such a critical system component? What were the thoughts of all the major distros when they decided to go with systemd then?

[dead]

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#189

Post by marginalia_nu »

d3Xt3r wrote: post #81
marginalia_nu wrote: post #38

Earlier quoted context omitted.

https://www.freedesktop.org/software/systemd/man/latest/syst... See StandardOutput= and StandardError=.

> Oh no! Bad Request > Error: access denied: error in challenge meta-refresh: mismatched token God I hate the modern web. I get that anti-bot measures are necessary, but at what cost?

You get the exact same information in

$ man systemd.exec

Though reading the question again, I should have probably linked to the equivalent of

$ man systemd-system.conf

as well, that's where you can set the default behavior across systemd, not per-service as the first man page is.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#190

Post by pineapplepizza6 »

marginalia_nu wrote: post #189
d3Xt3r wrote: post #81

Earlier quoted context omitted.

> Oh no! Bad Request > Error: access denied: error in challenge meta-refresh: mismatched token God I hate the modern web. I get that anti-bot measures are necessary, but at what cost?

You get the exact same information in $ man systemd.exec Though reading the question again, I should have probably linked to the equivalent of $ man systemd-system.conf as well, that's where you can set the default behavior across systemd, not per-service as the first man page is.

[dead]
Post reply on HN