Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

171–180 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#171

Post by microgpt2 »

otterley wrote: post #75
rcxdude wrote: post #70

Earlier quoted context omitted.

sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get…

Is that true even with PRAGMA journal_mode = WAL? https://www.sqlite.org/wal.html

Yes it's true even with journal_mode=WAL. The same pages are touched (minimum 1 full page plus one per index), only the atomicity layer is different.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#172

Post by microgpt2 »

TylerE wrote: post #137
giov4 wrote: post #128

Earlier quoted context omitted.

I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful. all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them. syslog experience? very efficient also on heavy load production instances,…

WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.

Gentoo also has a systemd-free option (and even if it didn't, you could make one because Gentoo is basically LinuxFromScratch)

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#173

Post by lyu07282 »

giov4 wrote: post #149
TingPing wrote: post #95

Earlier quoted context omitted.

You can trivially configure it to forward logs to another service to manage them.

that's not the point, why ship most common linux distros with an unreliable logging solution by default?

because its everything or nothing with systemd its a monolith wearing a trenchcoat

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#175

Post by pineapplepizza6 »

crabbone wrote: post #144
pengaru wrote: post #13

I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was…

I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right? On…

The principled excuse for mmap is when you're reading all over a file at high performance and you want to avoid either excessive syscalls or double caching. Which sounds like what a torrent program does but evidently it doesn't even work well for them.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#176

Post by pineapplepizza6 »

giov4 wrote: post #154
otterley wrote: post #45

Earlier quoted context omitted.

The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...

I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment... It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.

systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#177

Post by pineapplepizza6 »

p_l wrote: post #142
hedora wrote: post #71

Earlier quoted context omitted.

Or, you could write a plain text file. Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice. If you really ca…

Honestly, I would go append-only blocks that contain binary/compressed format with synchronizing marks so worst case you get some nulls but every reader can synchronize where they are in the stream without blocking anyone. Might take more space on disk than theoretical best of journald storage format with its absurd hashtables, but it fulfills the job of system log better and more complex format should be done in log…

You could even generate a bloom filter for every block, and when you have a full block of bloom filters, write them out to an index file.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#178

Post by pineapplepizza6 »

regularfry wrote: post #158
vachina wrote: post #103

Earlier quoted context omitted.

I just set the storage to volatile and the max log size to 10Mbytes and call it a day. Logs are really only useful at the tail.

Having the previous boot can get you out of a hole if you're dealing with dodgy drivers.

So change it back when you have a dodgy driver to deal with

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#179

Post by pineapplepizza6 »

shawnz wrote: post #28
otterley wrote: post #23

This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.

Designing a new on-disk format seems like a pretty far reaching architectural decision... I don't think that's an appropriate target for a drive-by fix from a new contributor

You could however choose to maintain a fork. If you had time. Few people do.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#180

Post by otterley »

brohee wrote: post #168
otterley wrote: post #77

Earlier quoted context omitted.

The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.

Writing multiple pages for a few hundred actual bytes also thrashes the OS cache...

Not if you use the correct cache hint flags on the write.
Post reply on HN