Live data from Hacker News

Avoiding Complexity with Systemd

mgdm.net

341–350 of 357 posts

Re: Avoiding Complexity with Systemd

#341

For me, systemd is the best thing since sliced bread. As a programmer, I now don't need to care about dropping privileges, managing logging, daemonization (the moment I need to do the double-fork dance again, chairs will be flying, I swear), dropping into a chroot, and do half-arsed guesses "is it up and running yet?" from a convoluted mess of shell code that looks like a bunch of hair stuck down a drain for a month.…

It's great... until something unexpected happens. Like your NIC doesn't have link on the ethernet cable and systemd waits for minutes without allowing you to abort waiting on the network because other units depend on it.

Or how about adding new buggy DNS code that doesn't work in common scenarios? Oh, sorry, here's another CVE because we didn't create enough test cases for the corner cases that are actually important.

Or oops, "nobody uses ntsysv", right? Or "We don't need to implement chkconfig even though we broke it".

systemd is a monolithic beast that is absorbing everything else in the system without considering that some of its decisions should be able to be disabled, and a lot of the design decisions are half baked. I don't believe in the philosophy of design its author has embraced. Progress is good, but please, stop breaking shit that has worked for decades. Anyone can write new code that partially implements a feature, but it takes real effort to responsibly migrate users from tools that worked to your new shiny half assed kitchen sink.

Re: Avoiding Complexity with Systemd

#342

Earlier quoted context omitted.

Isn't this something that, to a degree, might be mitigated by rate limiting journald has? IMO Lennart is right in that collection should be optimized to a point where it is (almost) never the bottleneck. I, personally, would put a filtering program between the chatty application and journald.

The scenario I personally have is: - Legacy application, with no access to source code. - Company that wrote the application hasn't existed for a decade. - It logs several useless lines every second. This is non-configurable, and includes a timestamp which makes each linue distinct. - It also logs some very important lines that _must_ be responded to. I don't think this is a unique situation to find yourself in, and…

How about to open this exceedly verbose legacy program with IDA and fill with NOPs all the places where this program logs something useless? I mean, this is an activity which really should be named as "hacking", in opposition to playing with init scripts.

Re: Avoiding Complexity with Systemd

#343
post #338

Earlier quoted context omitted.

I have never encountered the need for this and I honestly doubt you have either. Systemd fails mysteriously far less often than poorly written init files I have seen and it is honestly not that hard to debug almost every failure I've ever encountered. And I have worked on some very large scale systems with systemd. A systemd file I helped write has run many millions of times over without issue. I honestly find this c…

Ever worked on systems where you're on the hook for the guarantee that you can keep it running? We had 3 major breakages due to systemd, which is why we went back to something pretty much anyone who knows a bit of bash scripting can fix.

And you didn't have separate test and production sites?

Re: Avoiding Complexity with Systemd

#344
post #259

Earlier quoted context omitted.

Nope, it's still entirely incorrect. Again, the release team misjudged things. I was part of the GNOME release team at that time. We were actually warned about it, then misjudged it ("it'll be fine"). Further, it wasn't even a hard dependency. You're really not understanding components and APIs. > This forced other distros to either switch to systemd or drop support for GNOME. No, again entirely incorrect. GNOME runs…

Not GP. The fact that it took _work_ to get GNOME to run without systemd is a bad thing in my book. Also, to claim that systemd was selected on merit without anything backing up the claim of "merit" is disingenuous; plenty of worse solutions end up winning all the time.

It took work because systemd (more precisely logind) solved real problems and so the work had to be done again for the sake of non-systemd distros.

Re: Avoiding Complexity with Systemd

#345

Earlier quoted context omitted.

> I.e. sometimes computers just need to reboot, and there's nothing you can do about it. And this is the attitude that brings us shitty software, and "I dunno, just reboot to fix the problem?", which is what we have now. Short of kernel upgrades they really really don't. But if you've bought in to "oh computers just need to reboot sometimes", then I guess you fall into the category of people who have just given up on…

> And ALL of these services could in any case be fully started while you were typing your username and password. This is the key point you are refusing to hear. No, all of the services on a modern Linux machine can't be started while you're typing in your credentials. So they're started lazily, on-demand, one of the classic techniques for performance optimization and a hallmark of good engineering.

Of course they can. How many services do you they there are, installed, and how long do you think it takes to start them?

How long do you think it takes to start gpsd, or pcsd? Even my laptop has 12 CPU threads, all idle during this time. And including human reaction time (noticing that the login screen has appeared) this is, what, 10 seconds? 120 CPU-seconds is not enough? All desktops run on SSD now too, right?

In fact, how many services do you even think are installed by default?

And Linux, being a multitasking OS, doesn't even have just that window.

But you know, maybe it's a tight race. You could try it. How long does it take to trigger all those?

> a hallmark of good engineering.

In the abstract, as a "neat idea", yes. In actual implementation when actually looking at the requirements and second order effects, absolutely not.

You know you could go even further. You could simply not spin up the VM when the user asks to spin up a VM. Just allocate the IP address. And then when the first IP packet arrives destined for the VM, that's when you spin it up.

That's also a neat idea, and in fact it's the exact SAME idea, but it's absolutely clearly a very bad idea[1] here too.

So do you do this, with your VMs? It's cleary "started lazily, on-demand, one of the classic techniques for performance optimization and a hallmark of good engineering".

[1] Yes, very specific environments could use something like this, but as a default it's completely bananas.

Re: Avoiding Complexity with Systemd

#346
post #339

Earlier quoted context omitted.

> it's most useful with services where the startup time is less important than So now it's not about startup time at all? > please consider submitting bug reports and patches to help the project, Who says I don't? But systemd needs a few full time adults, not just a patch here and there to fix the launch-while-brainstorming culture that gave us the current situation.

It can also be about initial startup time of the system, not startup time of any individual service. Please follow the site guidelines, c.f. the part about "don't cross-examine" -- this is just a technical tool, I'm sure we both can come up with some ways that it could be useful, even if we wouldn't use them ourselves. >But systemd needs a few full time adults, not just a patch here and there to fix the launch-while-…

> Then start working it on it full time

I have told you multiple times why not.

Re: Avoiding Complexity with Systemd

#347
post #340

Earlier quoted context omitted.

> You keep saying this, without giving evidence to back it up I have, repeatedly. But like in this thread you just reply with the same question asked one more time: https://news.ycombinator.com/item?id=27653716 > People are still running Linux in memory constrained environments So why do they have all these memory-hungry services they don't need on standby? Does that mean that I can DoS these machines simply by conne…

I ask the same question because I haven't yet seen a better alternative that was given. If you have one, please show it, it would be very interesting to me. Otherwise, it sounds like you may not have that much experience with these tools, which is understandable. I can help find solutions, if you're interested. >Does that mean that I can DoS these machines simply by connecting to all the open ports, thus starting up…

> A lot of machines are not open to the public internet, so this probably doesn't apply there.

An internal audit is enough to trigger it. "Port scan crashes machine" is not exactly "reliable software".

> You can also use some cgroup managing tool (like systemd) to restrict memory usage to the process and configure the OOM killer behavior, so that would also prevent DoS attacks.

But that means that the default is bad, and unsuitable for resource constrained machines. Which circles back to "neat, but no actual use case".

> Actually no, this is wrong, at least for me when I tried the version of netstat that ships with debian. It only shows if something has the port open -- that thing could be an fd holding service

So you agree that it's a bad idea?

> So you're right that this complicates the system but this isn't really systemd's fault

It is, because it's needless complication. At least inetd was a model to make things simpler. It's the cgi-bin of network services.

But you'll notice that people don't write inetd-based services anymore. In fact my Ubuntu default install doesn't even have inetd installed.

> The only way to know for sure is to use a different tool that prints information about the owning process

netstat has supported this for (maybe) decades on Linux. It's the -p option.

But aside from systemd's poor choices if you see port 22 open, then you can actually be very sure that there is an sshd running, that successfully started (not too broken config).

You could still be wrong. Someone could have started netcat there, or just a honeypot, or whatever, but you can't tell me it's not useful information.

> Well now you got me confused, this seems to be directly conflicting with when you said this: "It will show you if you have an SSH server running"

… unless systemd broke this functionality. I'm making the point why it's a bad idea to break this.

Clients connecting will also not get useful error messages. Port is closed means service not running. Timed out waiting for SSH banner means something else.

Pre systemd it was essentially never anything other than inetd that held ports for others. And for about the last 20 years even it would only do things like echo,chargen,time service that people would run. And having those open by default is from a more naive time, where people thought "sure, why not run tftpd and time service, would could possibly go wrong?".

Nowadays they're off by default, because we're more experienced that any attack surface is still an attack surface, no matter how small.

Probably it helped that OpenBSD kept bragging about how many remote holes in the default install. It's not actually because OpenBSD had better code, it was just that a default OpenBSD only had OpenSSH open to the world.

Re: Avoiding Complexity with Systemd

#348

Earlier quoted context omitted.

Not GP. The fact that it took _work_ to get GNOME to run without systemd is a bad thing in my book. Also, to claim that systemd was selected on merit without anything backing up the claim of "merit" is disingenuous; plenty of worse solutions end up winning all the time.

It took work because systemd (more precisely logind) solved real problems and so the work had to be done again for the sake of non-systemd distros.

Can you tell me what problems logind solved? Because on my own machine, I don't have it, and I don't need it. In fact, it's a struggle to make sure it's not pulled in as a dependency of anything.

Re: Avoiding Complexity with Systemd

#349
post #339

Earlier quoted context omitted.

It can also be about initial startup time of the system, not startup time of any individual service. Please follow the site guidelines, c.f. the part about "don't cross-examine" -- this is just a technical tool, I'm sure we both can come up with some ways that it could be useful, even if we wouldn't use them ourselves. >But systemd needs a few full time adults, not just a patch here and there to fix the launch-while-…

> Then start working it on it full time I have told you multiple times why not.

Sorry I think I missed that. In that case you will have to find someone else who can do it and figure out how to get them paid. If you want help doing that, don't hesitate to ask.

Re: Avoiding Complexity with Systemd

#350
post #340

Earlier quoted context omitted.

I ask the same question because I haven't yet seen a better alternative that was given. If you have one, please show it, it would be very interesting to me. Otherwise, it sounds like you may not have that much experience with these tools, which is understandable. I can help find solutions, if you're interested. >Does that mean that I can DoS these machines simply by connecting to all the open ports, thus starting up…

> A lot of machines are not open to the public internet, so this probably doesn't apply there. An internal audit is enough to trigger it. "Port scan crashes machine" is not exactly "reliable software". > You can also use some cgroup managing tool (like systemd) to restrict memory usage to the process and configure the OOM killer behavior, so that would also prevent DoS attacks. But that means that the default is bad,…

>"Port scan crashes machine" is not exactly "reliable software". [...] So you agree that it's a bad idea? [...] But that means that the default is bad, and unsuitable for resource constrained machines.

I'm not sure I understand where you're coming from here? I explained how it could be made suitable, it could be done in a way that was crash resistant. I don't know if it's a bad idea or not, it depends on what you're trying to accomplish. The default here is configured by the distro, so you could expect to see a different default on an embedded distro.

>In fact my Ubuntu default install doesn't even have inetd installed.

I believe this is mostly because systemd has replaced its functionality.

>netstat has supported this for (maybe) decades on Linux. It's the -p option.

Good call, I forgot about that, I always use lsof. But that's exactly what I mean, it will show you which pid has the port open, so it will make it obvious if it's systemd or sshd. You won't be sure if there is actually an sshd running unless you check that. This really seems like a non-issue, you have all the tools you need to troubleshoot it.

>systemd broke this functionality. [...] Port is closed means service not running. [...] Pre systemd it was essentially never anything other than inetd that held ports for others.

I don't really want to discuss this anymore if I have to repeat myself, but this is not correct. There are multiple other reasons why you would have another service holding the fd open, such as load balancers, filtering proxies, userspace firewalls, etc, etc. The ability to pass an fd to a child process is an intentional feature of every Unix-like operating system that I've used. Systemd is only using the feature as the OS intended it, which is also supported on OpenBSD.

Post reply on HN