Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

71–80 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#71

Post by hedora »

quotemstr wrote: post #46
amluto wrote: post #36

Earlier quoted context omitted.

Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.

Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.

Or, you could write a plain text file.

Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice.

If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there.

In practice, no one wants that. It is strictly worse than returning partial entries after the hole, and by the time you are hitting this corner case, you are debugging a kernel crash.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#72

Post by otterley »

simoncion wrote: post #62
otterley wrote: post #34

Earlier quoted context omitted.

syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour... See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG... It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on…

> syslog doesn't have nearly the functionality that journald + journalctl does. A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks. Also, in regards to your original comment: > This issue report…

Nobody's talking about an obligation here. It's open source, and the maintainer owes non-paying users nothing. So if you want something fixed, it's now easier than ever to get involved in the fix. Nothing more, nothing less.

(Also, I'm not entirely sure this is a bug so much as an inefficiency report. Consumption of storage space isn't a documented or promised behavior, nor is the behavior technically incorrect. It's just wasteful.)

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#73

Post by hedora »

zbentley wrote: post #57
lokar wrote: post #26

For 99% of installs the basic assumption that local logging (with local reading) is the primary mode is just wrong.

What do you mean? That has described the vast majority of Linux systems I have ever touched, professionally or personally. Even corporate environments with log aggregation tail system logs rather than having them directly shipped elsewhere. The rare exceptions to this are some embedded devices without much durable storage, or tightly regulated environments in which log data is considered radioactive.

I’ve met people that think the log should be remotely stored and not written locally, since it’ll be shipped to splunk or whatever anyway.

Those people change their minds the first time a machine has intermittent network issues, and the logs needed to debug it are lost (or worse, the log buffer fills, then stdout fills, which backpressures the application, creating an outage while simultaneously eating the logs).

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#74

Post by otterley »

marginalia_nu wrote: post #64
otterley wrote: post #34

Earlier quoted context omitted.

syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour... See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG... It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on…

What is the real-world use for these features? Who is this built for? Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.

It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#75

Post by otterley »

rcxdude wrote: post #70
throw0101a wrote: post #63

Earlier quoted context omitted.

I was always curious why they created their own format rather than leveraging SQLite, OpenLDAP's LMDB, etc .

sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get…

Is that true even with PRAGMA journal_mode = WAL?

https://www.sqlite.org/wal.html

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#76

Post by rcxdude »

otterley wrote: post #74
marginalia_nu wrote: post #64

Earlier quoted context omitted.

What is the real-world use for these features? Who is this built for? Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.

It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.

The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#77

Post by otterley »

rcxdude wrote: post #76
otterley wrote: post #74

Earlier quoted context omitted.

It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.

The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.

The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#78

Post by marginalia_nu »

otterley wrote: post #74
marginalia_nu wrote: post #64

Earlier quoted context omitted.

What is the real-world use for these features? Who is this built for? Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.

It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.

I did, and I still don't get why you would want this monstrosity over a structured append-only log file. If you want to index the data, you can do that when you roll over the file. That way you get the exact same robustness guarantees, without the insane architecture.

Like ultimately it isn't even fast, journalctl is so bad at rendering text that it's approximately still as slow as seeking in a 400 MB .log-file using less.

Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#79

Post by otterley »

marginalia_nu wrote: post #78
otterley wrote: post #74

Earlier quoted context omitted.

It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.

I did, and I still don't get why you would want this monstrosity over a structured append-only log file. If you want to index the data, you can do that when you roll over the file. That way you get the exact same robustness guarantees, without the insane architecture. Like ultimately it isn't even fast, journalctl is so bad at rendering text that it's approximately still as slow as seeking in a 400 MB .log-file using…

> Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.

That I agree with. I don't personally use journalctl much these days, particularly now that practically everything's a container and all their logs are getting shipped off-host for indexing. But I get why, 14 years ago, it was considered a good idea.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#80

Post by 0x_rs »

journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its dumping ground. To be fair, the kernel and usb peripherals can also have a bad day and spam 3 million lines an hour into it, think input irq status -75.

It's too much of a chore to keep up with all the program-level configs (if they have them) and service files, but LogFilterPatterns in systemd can help in an unintended way: you can make one log blacklist with a .conf file in /etc/systemd/system/service.d/, and put in there all the patterns that spam your journal one by one, don't even have to chase misattributed loglevels. It just looks something like:

[Service]

LogFilterPatterns=~I am a completely useless log entry

LogFilterPatterns=~I am another useless log entry

But it doesn't pick up on identifiers and doesn't do anything for kernel spam. It's only great to make some messages shut up. Also, I'd consider any btrfs install that does not have nocow on cache, journal etc. to be defective.

Post reply on HN