Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

61–70 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#61

Post by ericpruitt »

otterley wrote: post #33
lucb1e wrote: post #24

Earlier quoted context omitted.

Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component

Sure, a concrete proposal first would be a good idea. That said, would you look a gift horse in the mouth?

Because you become responsible for feeding and taking said care of horse and dealing with any technical debt associated with it. If someone submits code to a project that I maintain that's going to make my life difficult in the future, I'm not going to accept it.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#62

Post by simoncion »

otterley wrote: post #34
deepsun wrote: post #31

Earlier quoted context omitted.

Well, they are comparing in comments with syslog, and it does better, as you asked. Syslog was there for 46 years.

syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour... See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG... It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on…

> syslog doesn't have nearly the functionality that journald + journalctl does.

A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks.

Also, in regards to your original comment:

> This issue report feels like it ought to be accompanied by a fix.

This smells a lot like the "Don't come to me with problems, come to me with fixes." order that a lot of mid-level and director-level management really loved to make five, ten years back. [0] While this sounds like a hard-charging order and gives the impression that it's bringing much-needed discipline to lazy-ass subordinates, the truth of the matter is that its actual effect [1] is to get people to shut the fuck up about the company's problems. The job of most mid-level and nearly all director-level management is to do inter-organization coordination. Most low-level folks don't come to mid- or director-level management with problems they can solve. After all, if they could solve them, they would... talking to folks in that layer of management is usually a huge drag. Most low-level folks only come to these sorts of folks with issues that require inter-organization coordination!

So, yeah... the only obligation of someone who's reporting a bug is to provide a reasonably well-written bug report accompanied with reproduction instructions and diagnostics that are as clearly written as is reasonably possible. Reporters of performance bugs are under no obligation to suggest how to eliminate the bug... especially not if the project they're reporting the bug against has both paid maintainers and claims it's the infrastructure on top of which all Linux systems should be built. Corporate-backed projects that make such grand claims put themselves in a radically different class than the one that covers hobby or small-time projects.

[0] AIUI, it came out of Google, but my understanding might be incorrect.

[1] ...regardless of whether or not that effect is intentional...

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#63

Post by throw0101a »

simoncion wrote: post #59
otterley wrote: post #39

Something must have happened along the way, because this was not the original design intent of the database (emphasis mine): """ The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the n…

If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality. Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped…

I was always curious why they created their own format rather than leveraging SQLite, OpenLDAP's LMDB, etc.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#64

Post by marginalia_nu »

otterley wrote: post #34
deepsun wrote: post #31

Earlier quoted context omitted.

Well, they are comparing in comments with syslog, and it does better, as you asked. Syslog was there for 46 years.

syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour... See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG... It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on…

What is the real-world use for these features? Who is this built for?

Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#65

Post by simoncion »

throw0101a wrote: post #63
simoncion wrote: post #59

Earlier quoted context omitted.

If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality. Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped…

I was always curious why they created their own format rather than leveraging SQLite, OpenLDAP's LMDB, etc .

Given that the ethos of the Systemd Project is to almost-always reinvent so that they retain complete control [0] over the code, I'm never surprised when they choose to reimplement and fold that implementation into the project rather than to cooperate and improve the state of the world for all projects.

[0] As demonstrated by many vertically-integrated successful businesses -SpaceX being a recent example- there are substantial benefits to doing everything in-house. However, if you choose to pull an assload of things in-house to do them yourself, you must be capable of doing all of that work yourself. Given the state of SystemD, [1] its historical and current reaction to reports of both subtle but severe bugs and of totally reasonable system configurations that SystemD makes impossible, I don't believe they have the capability required to do a good job at what they've set out to do. Choosing to not cooperate with the existing ecosystem was a short-term win, but -IMO- a huge long-term mistake.

[1] ...this is spelt "SystemD" not as a slur, but to distinguish systemd(1) from The Systemd Project it is a part of. It's damn annoying that they share the same name...

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#67

Post by hedora »

Someone should implement a new operating system that can efficiently handle text processing.

It could have some simple tools that let you generate reports, display them on screen, and compose tools for that sort of thing in a natural way.

We could call it UNIX.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#68

Post by sidewndr46 »

rasz wrote: post #22
pudgywalsh wrote: post #12

How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly? The first thing I do on a Linux system is install a proper syslog daemon.

One of the first things I do on win10 is disable most of excess logging.

How do you do that?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#69

Post by hedora »

smartmic wrote: post #3

I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.

I’ve been using devuan more or less since day one. I highly recommend it.

FreeBSD isn’t too shabby these days either.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#70

Post by rcxdude »

throw0101a wrote: post #63
simoncion wrote: post #59

Earlier quoted context omitted.

If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality. Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped…

I was always curious why they created their own format rather than leveraging SQLite, OpenLDAP's LMDB, etc .

sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get complaints about logs being lost during a power failure or kernel panic)
Post reply on HN