Avoiding Complexity with Systemd • Hacker News Board

Re: Avoiding Complexity with Systemd

#321

Post by knorker » Mon, Jun 28, 2021, 1:25 PM UTC

throw-8462682 wrote: post #319

knorker wrote: post #316

Earlier quoted context omitted.

> We need desktops to boot up fast, because you said it yourself, sometimes they just need to I didn't say that. Because they don't. > And no one likes waiting around for their machines to boot. Nobody cares, if it's once for every month or two. Which it is. > we need servers to boot up fast But it's not actually booted until the service is up, so it's moot.

> But it's not actually booted until the service is up, so it's moot. With per second billing, fast boot times save money and enable lower fixed capacity, further lowering cost.

But no it doesn't. Until your service is started, your service is NOT actually booted. That's what I said.

You are not paying per-second for the VM. The VM itself adds zero value to you. It's the service that's running (or in this case, not) that you're paying for.

Who cares how long it takes before systemd calls listen()? Nobody derives value from that. You're not paying for that. You're paying for the SERVICE to be ready. And if you're not, then why are you even spinning up a VM, if it's not going to run a service?

It's just clever accounting.

Re: Avoiding Complexity with Systemd

#322

Post by josephcsible » Mon, Jun 28, 2021, 2:20 PM UTC

kaba0 wrote: post #295

josephcsible wrote: post #292

Earlier quoted context omitted.

Are you seriously claiming that processes ignoring or handling signals rather than dying is an ill defined hack? That's a well defined, long established, normal way of doing things on UNIX and Linux systems. Also, that really isn't a good way of detecting frozen processes, since in your world, the only answers you'd ever get are "it's frozen" or "it wasn't frozen and now it's dead". > They can easily register a user…

> Are you seriously claiming that processes ignoring or handling signals rather than dying is an ill defined hack? Yes. Just because it is 50-years-old doesn’t make it suddenly a valid approach. There are 4 states here - process running with usual semantics for closing, process running that would like to linger in the background, and the frozen version of this two. How should a service manager/resource handle — that…

What ways are there to tell if a process is frozen that only work if it registers a systemd user service?

Re: Avoiding Complexity with Systemd

#323

Post by citrin_ru » Mon, Jun 28, 2021, 2:35 PM UTC

geofft wrote: post #104

einpoklum wrote: post #100

Earlier quoted context omitted.

1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. 2. Some of the systemd logging is binary, so good luck with that if there's a problem. 3. Have you tried non-systemd init systems other than sysvinit? 4. Yes, it is convenient when everything below your development is cenrtalized by a single entity. It can easily provide a consistently useful underpinning. But…

I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip (Not to ment…

1. journalctl is painfully slow. Most frequent commands I use: journalctl -u myunit --since=today and journalctl -u myunit -n 1000 take minutes on loaded servers with large amount of logs. 1st command is instant for daily rotated plain text logs and 2nd is almost instant for any plain text logs.

In theory it is possible to make a binary log database which will work fast for many queries including these, but journalctl is not fast at all.

2. People are fine with gzip because log compression is usually done in a way which makes log corruption/loss highly unlikely:

gzip log.ystd, fsync log.ystd.gz, rm log.ystd

At any time you can reset a server and you will either have uncompressed or compressed logs, not some partially written binary blob.

Re: Avoiding Complexity with Systemd

#324

Post by zxzax » Mon, Jun 28, 2021, 3:22 PM UTC

nine_k wrote: post #311

zxzax wrote: post #310

Earlier quoted context omitted.

To me it's the other way around -- pipewire is the sound server equivalent of systemd. For better or for worse.

What would you prefer then? Jack?

I just use systemd and pipewire, it's not a problem for me personally. They're the worst options... except for all the others :)

Re: Avoiding Complexity with Systemd

#325

Post by zxzax » Mon, Jun 28, 2021, 3:25 PM UTC

knorker wrote: post #317

zxzax wrote: post #212

Earlier quoted context omitted.

You can just disable the fd passing for services that don't need it. I'm not sure what your actual issue is. I haven't had to reboot any more with systemd than I did with sysvinit or openrc, or the slackware rc init, or anything else really. If you have an actual crash that is causing you problems, you should consider reporting it or submitting a patch to fix it, just like you would with any other open source that yo…

> You can just disable the fd passing for services that don't need it. I'm not sure what your actual issue is. First of all this just sounds like "I don't know what your problem with systemd is, you can just choose to not use it". Second, my point is that it's an absolutely terrible idea, and should never have been done. > I believe systemd was inspired by other established unix service managers like macOS launchd an…

If you think the socket activation takes too long you can just turn it off and start that service unconditionally. What's the problem? Other service managers support this too, it's not a systemd only thing. Maybe it's a bad idea for some services but others would seemingly disagree that it's always a terrible idea, and not just those who are systemd developers. You may just be using it for the wrong services -- it's most useful with services where the startup time is less important than reducing the overall memory usage on the system.

If you have experience making software reliable, please consider submitting bug reports and patches to help the project, like you would do with any other open source that you depend on. I'm sure contributions to improve the testing and CI would be appreciated. A high school student can also trash talk loudly about things they didn't take the time to fully understand (and I admit I did a lot of that when I was a teenager in high school), but it takes real expertise to illustrate what the actual problem is and to contribute a fix for it in a positive way.

Re: Avoiding Complexity with Systemd

#326

Post by dec0dedab0de » Mon, Jun 28, 2021, 4:19 PM UTC

zlynx wrote: post #280

dec0dedab0de wrote: post #239

Earlier quoted context omitted.

Say the computer writing the log was destroyed, you were able to recover the storage, but the only computer available did not have journalctl. Or if you're writing to network storage and would like to analyze the logs from your haiku box. Text is not perfect, but it's the one thing that is always available. Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption we…

The journalctl binary format seems to handle corruption pretty well. That was a design criteria. Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT. It is encoded in 7-bit ASCII with records delimited by 0x0a bytes. Corruption tends to be missing data, and so the reader has to jump ahead to find the next synchronization byte, aka 0x0a. This also leads to log parsers producing complete trash as th…

The journalctl binary format seems to handle corruption pretty well. That was a design criteria.

Thanks for pointing that out. I guess thats why they came up with their own format instead of just using sqlite, or something else that is already a standard.

Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT

That's a bit pedantic, even for HN standards :-)

But yes, I know all about fragile log parsers and race conditions of multiple processes writing to the same file. I was just thinking about a scenario where you end up having to read raw logs when things go haywire.

Re: Avoiding Complexity with Systemd

#327

Post by dec0dedab0de » Mon, Jun 28, 2021, 4:27 PM UTC

geofft wrote: post #271

dec0dedab0de wrote: post #239

Earlier quoted context omitted.

Say the computer writing the log was destroyed, you were able to recover the storage, but the only computer available did not have journalctl. Or if you're writing to network storage and would like to analyze the logs from your haiku box. Text is not perfect, but it's the one thing that is always available. Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption we…

> the only computer available did not have journalctl This is exactly what I don't understand. This is a world where no other computer exists? I have bigger things to worry about (even if you scope the problem down to to "no other computer with journalctl installed exists on my network"). > Or if you're writing to network storage and would like to analyze the logs from your haiku box. I don't have any Haiku boxes, bu…

This is exactly what I don't understand. This is a world where no other computer exists?

What about a windows user that has a smart device of some sort that is acting up. They're able to dump the files, but now they have to read them. It's a more realistic scenario than a haiku box, but its the same idea.

And for the record, I am against gzip logs too. I think if you need to save that many logs you should be exporting them to another system instead of archiving them locally.

Re: Avoiding Complexity with Systemd

#328

Post by zxzax » Mon, Jun 28, 2021, 4:37 PM UTC

knorker wrote: post #320

squiggleblaz wrote: post #312

Earlier quoted context omitted.

> Here's an example: Someone read that fd-passing is a thing, so now systemd listens to just about everything and spawns stuff on-demand. This feature benefits some people. Even if your service requires socket activation, you can still set the service to startup and you can still control it with normal command lines. > It's a great idea, if you're in high school. You make this accusation twice, but you give no-one an…

> This feature benefits some people. Even if your service requires socket activation, you can still set the service to startup and you can still control it with normal command lines. But it's broken by design. It's a bad idea. It's "neat", but "neat" doesn't add value. > You make this accusation twice, but you give no-one any reason to believe it (unless you think "using an optional feature I don't want to use and wh…

>But it's broken by design. It's a bad idea.

You keep saying this, without giving evidence to back it up. People are still running Linux in memory constrained environments, those didn't go away now that the 90s are over.

>Using this feature pushes orders of magnitude of complexity onto users, and greatly reduces the ability to error handle or even know the status of services.

To be clear, it sounds like what you're suggesting is that these services implement their own fd holding logic, which is going to be even more complex, and is exactly what systemd is trying to prevent from happening.

>You check if the service is turned on (systemctl or whatever), you check if the process is running (ps, etc), and you check listening ports (netstat, nmap, etc...), they all agree that the service either is or isn't running. And if it's running, it's successfully run its initialization and should be usable.

This isn't really correct, netstat or nmap won't show process status at all. You really don't know what the real status of that port is unless you've run lsof or something else that scans the open fds of the processes, and such a tool would make it obvious when systemd (or some other fd holder) has the fd open. Also, systemctl will display this separate socket/service units, so you can just check if the socket unit is running but not the service.

Re: Avoiding Complexity with Systemd

#329

Post by nh2 » Mon, Jun 28, 2021, 4:47 PM UTC

Arnavion wrote: post #285

nh2 wrote: post #269

Earlier quoted context omitted.

> Why does this "not compose well" ? Because you cannot easily write libraries/components that do this. In NixOS, other modules can override the options of other modules. For example, a a web app can set the nginx options that it needs, instead of requiring you (the admin) to "drop a file" in /etc. This is one of the reasons why on Ansible Galaxy (community repository of Ansible roles) there are 527 nginx roles [1],…

What does systemd.services.nginx.unitConfig.StartLimitIntervalSec = 0; do that it doesn't require root? If the point is that it's not manipulating a system-level nginx service but a user-level one, then writing the systemd override file in the way I described doesn't require root either.

Applying the NixOS config requires root -- that's not what it's about.

What I mean is that services can set other services' options, without you (the admin) having to to write such overrides manually.

Re: Avoiding Complexity with Systemd

#330

Post by yawaramin » Mon, Jun 28, 2021, 5:12 PM UTC

knorker wrote: post #315

yawaramin wrote: post #265

Earlier quoted context omitted.

> but you CAN'T, because some idiot decided that it'll only actually start when someone connects to its unix socket, so running "restart" is a no-op. FTA: > It can then boot your service on the first request, or you can do systemctl start lunchd yourself if you think that would take a while.

You can, but it doesn't. If I know I want to start it I can just connect with the client. Sounds like you're arguing something like "yes, it's broken. But you can just reimplement everything and it won't be broken anymore". Yes, I can also turn off "kill user processes on logout", but that doesn't make that not-broken.

No, I'm arguing there's a way to force-start even socket-activated services.

But this is really a moot point. Systemd's socket activation is really meant for system services which would otherwise be in the critical path of system boot. 'Regular' client-facing services that people normally run–webapps, etc.–are not really the target use case. It's fine to start them up in the normal way, with WantedBy=multi-user.target in the [Install] section. And I have never seen people use socket activation for them anyway. So you are basically arguing a strawman here.