Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

github.com

Post reply on HN
Print view View original

201–210 of 239 posts

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#201

Post by stryan »

magicalhippo wrote: post #94
barrkel wrote: post #8

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's bet…

Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald. Why hasn't this been done if it's that terrible?

While the rest of systemd is actually pretty modular, journald is unfortunately the only other required component. You can not run systemd without journald running in some way; closest you can get is setting Storage=none and forwarding the logs elsewhere.

journald is in a weird state where its "good enough" and mandatory that most people forget how bad it is until something like this pops up.

Normally I'm pretty happy with systemd and its many components; I even willingly run systemd-resolved, which is probably the other most hated component. But journald makes a lot of weird choices and if I could drop it I would in a heartbeat.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#202

Post by zbentley »

speed_spread wrote: post #161
zbentley wrote: post #56

Earlier quoted context omitted.

Say more? Sounds like a good story

My caveman understanding is that mmap writes are bad for transactional accesses because you have little to no control over sync. The OS can decide to commit changes to disk anytime, in any order which is the opposite of what you want for anything ressembling a database.

By default yes, that’s true. But while there isn’t a reliable don’t-flush-this-page system, there definitely are ways to force the flush of specific ranges in an mmapped file.

But you’re generally right. I think that’s why most databases have the notion of a WAL, which is carefully append-only. But the non-WAL data files in most DBs I’ve used are accessed via mmap.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#203

Post by simoncion »

otterley wrote: post #97
simoncion wrote: post #96

Earlier quoted context omitted.

> Please respond to the strongest plausible interpretation... Assume good faith. That's what I did. So, right back at you.

> That's what I did Please explain, because it's not coming across that way. It's coming across as needlessly picky and combative, especially after I told you what I meant (or, at least, didn't mean) and you continued to argue with me.

> Please explain...

I'm neither required nor strongly obligated to do so, nor do I see significant personal benefit to doing so. So, I will not.

However, these days it's quick and easy to command an LLM-based system to generate most any text. Before one demands an explanation from a human, perhaps one should machine-generate a plausible-sounding explanation and present that along with one's demand for a human-synthesized one?

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#204

Post by lokar »

zbentley wrote: post #195
lokar wrote: post #108

Earlier quoted context omitted.

By count, most installs will be the large cloud providers

Agreed. And in my experience , most large cloud providers’ Linux systems I’ve worked on (either their VMs as a tenant or their underlying hardware as an employee) log locally and ship additionally.

We did a small amount of system logs, but anything high volume went directly remote

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#205

Post by simoncion »

eviks wrote: post #129
emmelaich wrote: post #104

Earlier quoted context omitted.

"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.

Yes, a lot of awful practices have always been a thing. > e.g. hook in a senior engineer that you know is intimate with the system. unless, of course, you don't know said engineer because you don't even work in the same company, you're a just a user seeing a problem in an app you use

> unless, of course, you don't know said engineer because you don't even work in the same company...

Or they work in a different part of the fairly-large company that you both work for.

I guess emmelaich either missed the part of my commentary where I talked about handling inter-organization communication, and/or has never worked at a company where it's simply impossible to know everyone who could reasonably be relevant to the stuff that the company works on.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#206

Post by otterley »

itvision wrote: post #148
sidewndr46 wrote: post #66

years ago, I set Storage=volatile on almost all the journalD configurations I have. This largely solved this kind of problem.

Good for your personal devices, very not good for servers where you need to have any sort of accountability and security trail.

That’s why you ship the logs off host to a central collector.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#208

Post by otterley »

simoncion wrote: post #203
otterley wrote: post #97

Earlier quoted context omitted.

> That's what I did Please explain, because it's not coming across that way. It's coming across as needlessly picky and combative, especially after I told you what I meant (or, at least, didn't mean) and you continued to argue with me.

> Please explain... I'm neither required nor strongly obligated to do so, nor do I see significant personal benefit to doing so. So, I will not. However, these days it's quick and easy to command an LLM-based system to generate most any text. Before one demands an explanation from a human, perhaps one should machine-generate a plausible-sounding explanation and present that along with one's demand for a human-synthes…

facepalm

Way to double down on the “needlessly picky and combative” angle, dude.

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#209

Post by pengaru »

quotemstr wrote: post #132
pengaru wrote: post #127

Earlier quoted context omitted.

When did this become a discussion limited to journald on btrfs? and that seems like something btrfs should fix at some point

So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point. > When did this become a discussion limited to journald on btrfs? btrfs is in the HN thread title. > and that seems like something btrfs should fix at some point…

> btrfs is in the HN thread title.

as is ext4

Re: Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

#210

Post by simoncion »

otterley wrote: post #208
simoncion wrote: post #203

Earlier quoted context omitted.

> Please explain... I'm neither required nor strongly obligated to do so, nor do I see significant personal benefit to doing so. So, I will not. However, these days it's quick and easy to command an LLM-based system to generate most any text. Before one demands an explanation from a human, perhaps one should machine-generate a plausible-sounding explanation and present that along with one's demand for a human-synthes…

facepalm Way to double down on the “needlessly picky and combative” angle, dude.

> Way to double down on the “needlessly picky and combative” angle, dude.

Sit and consider the points of similarity between my refusal-shaped reply and the entire conversation we had prior to it and you might find enlightenment, in the style of those classic Zen tales. Perhaps an LLM-based tool might be able to assist you in this, or maybe it will be distracting and misleading.

GLHF and all that.

Post reply on HN