161–170 of 239 posts
Ooh, mmapped writes. I make that mistake once, years ago. :) I posted a comment in that GH issue.
Say more? Sounds like a good story
Something must have happened along the way, because this was not the original design intent of the database (emphasis mine): """ The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the n…
This optimally solves the problem of the space taken by logs on disk.
The only possible disadvantage is that any application that is used to scan the logs must decompress them, but in practice I have never seen any case when this caused any nuisance, even when using such a primitive solution like "zcat|grep", instead of a full-featured application.
Earlier quoted context omitted.
Text or text-like (e.g. text content with simple control char delimiters for metadata) would be far superior than the slow-down from Sqlite's safety mechanisms. Optimising logs for read, at the expense of write, is a bad pattern to me.
Read optimized? That is funny because reading logs from journald is dog slow compared to, you know, log files.
I completely agree with one of the comments from there: > But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better. It really does not make any sense to use memory-mapped files when writing logs. Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files. Only when rea…
great and clear summary thank you! I would laso add that if my design decisions or development actions lead to an issue affecting multiple linux distro defaults I would feel responsible and rush for a solid fix instead of this https://github.com/systemd/systemd/issues/15292#issuecomment...
Something must have happened along the way, because this was not the original design intent of the database (emphasis mine): """ The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the n…
The traditional solution for the problem of the repetitive data included in logs is that every time when a log file grows over a certain size (or periodically in time), a new log file is created and the old file is compressed with some standard data compression algorithm, which eliminates the repetitions. This optimally solves the problem of the space taken by logs on disk. The only possible disadvantage is that any…
If decompression is a pain though, change your logrotate so it doesn’t compress. Obviously costs more in disk space and less in compute.
Earlier quoted context omitted.
I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful. all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them. syslog experience? very efficient also on heavy load production instances,…
WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.
Earlier quoted context omitted.
The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment... It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.
Earlier quoted context omitted.
The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.
The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.
Something must have happened along the way, because this was not the original design intent of the database (emphasis mine): """ The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the n…
The traditional solution for the problem of the repetitive data included in logs is that every time when a log file grows over a certain size (or periodically in time), a new log file is created and the old file is compressed with some standard data compression algorithm, which eliminates the repetitions. This optimally solves the problem of the space taken by logs on disk. The only possible disadvantage is that any…