Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A timesyncd failure and systemd's lack of debugability

utcc.utoronto.ca

Post reply on HN
Print view View original

71–80 of 126 posts

Re: A timesyncd failure and systemd's lack of debugability

#71

Post by mrweasel »

The more I use Systemd the more I actually like it. It has some weirdness to it and there are parts that aren't really useful.

Journald is pretty stupid when you think about it. Either you're setup is really small, and logging to text files by defaults would be simpler to use. Alternatively your setup is large enough that you have a log-host/log-analytics, and logging to text files would be a sane default, as you need to convert the binary log to text anyway to do log-shipping.

It hard to argue that writing startup script hasn't improved. I would much rather write a service file than an up-start script.

Re: A timesyncd failure and systemd's lack of debugability

#72

Post by majewsky »

iforgotpassword wrote: post #61

I'm repeating myself, but the best way to use systemd is as an init system only, and then staying away from those fancy features like dynamic users (wtf) or socket activation. Especially avoid all that bolted on crap like timesyncd, resolved, etc. Yes they integrate a bit better with systemd, but no you don't need this better integration, especially since these tools are usually less configurable than their bigger br…

> Especially avoid all that bolted on crap like timesyncd, resolved, etc.

You're pretty much stuck with systemd-resolved when you use systemd-networkd.

And systemd-networkd is a godsend. It's the first network setup tool that I've found that is:

- full-featured

- simple to set up for the 99% usecases like "just do DHCP whenever a physical interfaces goes UP"

- available and works the same on all major distros

- not a Lovecraftian monstrosity like network-manager

Re: A timesyncd failure and systemd's lack of debugability

#73

Post by hvidgaard »

thaumaturgy wrote: post #34

Let's carefully step around the whole distraction that is systemd. Instead: > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. This see…

> > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it.

> This seems to be really common practice in modern software development, and it sucks

I have this discussion with the PM and CEO every now and then. It is impossible to give meaningful error messages for all errors. For known errors you can give a meaningful error message, but for errors not yet discovered it's more luck than anything if you give a meaningful error message. The best you can do it make the software robust so it'll work if a non critical subsystem/plugin fails.

The next part of the discussion is usually them suggesting to just allocate time to figure most errors and write meaningful error messages for them. But the problem is that this isn't a mechanical system, but a system several orders of magnitude more complex, and even if we did spend multiple man-years doing it, the cost in terms of added complexity would be enough to get any management directly responsible fired.

That said, I completely agree that developers should spend time supporting and running systems so they understand the impact of some of those decisions. Especially should it be expected that when you encounter errors that aren't bugs in the source code, you account for them.

Re: A timesyncd failure and systemd's lack of debugability

#74

Post by majewsky »

fooker wrote: post #53
ploxiln wrote: post #43

Earlier quoted context omitted.

With systemd, any really tricky low-level bug is systemd's fault, because systemd tries to do everything, in the most "advanced" way. And due gnome3 dependencies it's been adopted by all major distros. Before systemd, a difficult bug was not always the fault of one newfangled opinionated and unavoidable package. I read the first blog post about systemd when it was written, and thought it was really interesting. Then…

>And due gnome3 dependencies it's been adopted by all major distros. That's not correct. It has been adopted by all major distros due to ease of use and maintaibility of unit files. That is very clear from nearly all mailing list discussions. Also, people can and do use gnome3 without systemd. ( https://wiki.gentoo.org/wiki/GNOME/GNOME_Without_systemd ).

> That's not correct. It has been adopted by all major distros due to ease of use and maintaibility of unit files.

Or rather, because unit files work the same across all systemd-using distros, they can actually be upstreamed to the application developers and distros just package them, which is a huge time-saver compared to having to maintain init scripts for hundreds of daemons.

Re: A timesyncd failure and systemd's lack of debugability

#75

Post by gnud »

andrewstuart wrote: post #6

Before systemd, a difficult bug was a difficult bug. With systemd, a difficult bug leads to an attack on systemd itself, and questioning whether it should even exist. I like systemd, it's a pity some vocal people aren't enthusiastic about it. Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking…

You obviously didn't read the article before deciding this wasn't systemd's fault.

The bug is the lack of any error handling in the DynamicUser feature of systemd. The original failure is not logged anywhere - and launching timesyncd just goes ahead even though the setup for it fails completely. Naturally timesyncd then fails to start.

(Or perhaps the bug is the whole DynamicUser feature. From reading the bug report, it looks like it will fail any time anyone has a user-only FUSE mount point on their system...)

Re: A timesyncd failure and systemd's lack of debugability

#76

Post by crististm »

andrewstuart wrote: post #6

Before systemd, a difficult bug was a difficult bug. With systemd, a difficult bug leads to an attack on systemd itself, and questioning whether it should even exist. I like systemd, it's a pity some vocal people aren't enthusiastic about it. Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking…

Systemd is the light that people don't want to look at and embrace, heh?

What I see is a an appalling amount of arrogance that comes with the "I'll solve all your current and future problems".

Systemd actually doesn't and in fact can't do what it pretends. The evidence is in the never ending list of posts like these that pop up all the time. And the arrogance of pretending to be the top of the init.d hierarchy of init scripts is not sustained by the competence of actually fixing the problems that people keep reporting.

There is a reason for the existence of the systemd opposition, and is similar to the opposition you find in any field where the top of the hierarchy is abusively taken by force instead of by competence. This in fact will be the reason for systemd downfall and I'm confident to say that it will take with it everything that sustained it in it's position.

Re: A timesyncd failure and systemd's lack of debugability

#77

Post by fooker »

majewsky wrote: post #74
fooker wrote: post #53

Earlier quoted context omitted.

>And due gnome3 dependencies it's been adopted by all major distros. That's not correct. It has been adopted by all major distros due to ease of use and maintaibility of unit files. That is very clear from nearly all mailing list discussions. Also, people can and do use gnome3 without systemd. ( https://wiki.gentoo.org/wiki/GNOME/GNOME_Without_systemd ).

> That's not correct. It has been adopted by all major distros due to ease of use and maintaibility of unit files. Or rather, because unit files work the same across all systemd-using distros, they can actually be upstreamed to the application developers and distros just package them, which is a huge time-saver compared to having to maintain init scripts for hundreds of daemons.

Makes sense.

Re: A timesyncd failure and systemd's lack of debugability

#78

Post by fowl2 »

lomnakkus wrote: post #58
erik_seaberg wrote: post #5

systemd's job is to start daemons. It can be reasonable to decide dynamically when to start them based on which others are healthy (maybe you care more about speed than determinism on a laptop). But there's no reason to choose a uid and mkdir dynamically. You want these errors at install time, not at the box's next (probably unattended) restart.

Personally, I think even having to choose a UID at install time or dynamically is a pretty sad indictment of just how broken the user system on Linux (and pretty much every UNIX-like) is. It leads to a huge amount of incidental complexity where there's a much simpler way: User IDs should just be randomly chosen UUIDs -- either generated by upstreams or by the distros. Any metadata attached to the user UID could just…

Except of course if you want multiple instances of the same daemon

Re: A timesyncd failure and systemd's lack of debugability

#79

Post by tannhaeuser »

dijit wrote: post #26
cwyers wrote: post #25

Earlier quoted context omitted.

Distro maintainers didn't force you to do anything. They made a decision about how to allocate their limited resources. And we're past the days of Linux being a primarily volunteer OS, but a fair proportion of distro maintainers are still volunteers. They are not at your beck and call. If you want to have a Linux distro without systemd, there's several. If none of them meet your needs, pitch in with time or money to…

False dichotomy. I pay redhat a lot of money, short of paying people full time to maintain a mirror distro of redhat without systemd, which would be impossible given how embedded it has become into everything. Some decisions cannot be unmade. Short of starting from scratch with gentoo (which is a Herculean effort at scale) I’m not sure what you’re saying. I cannot be disgruntled at choices because distro maintainers…

> maintain a mirror distro of redhat without systemd, which would be impossible given how embedded it has become into everything

Not RH-based, but have you checked out Devuan (systemd-free Debian)? I've always valued RHEL's (up until 6) stability, but I'm seeing Debian/Ubuntu becoming more and more used in enterprise roles where before these had already been strong in web roles. I'm also seeing a need for a systemd-free minimal O/S for container (not container host) roles.

Re: A timesyncd failure and systemd's lack of debugability

#80

Post by nullify88 »

zaarn wrote: post #66
KaiserPro wrote: post #37

Earlier quoted context omitted.

Before systemd the error were shell errors. before systemd the bundled logger wrote to a file, didn't direct people to journald, that lags like a champ when a log file is > 1 meg. before systemd errors in config files were on specific lines I don't have bias against systemd I have bias against it's flaws. I like its config file. I hate it's documentation. I like that it has a cron like feature, I hate that its pretty…

sysvinit has it's warts too and it can get quite ugly. For example, atm I need to have 3 services. Two of these services need to run at the same time, they can run separately but then they are not useful. The third services can only run with both services online and is first run 15 minutes after boot (not earlier!) and then every hour but the hourly run MUST NOT run before the first run of the boot (ie the very first…

I probably would have resorted to another process manager if I were on sysvinit such as supervisord, god, monit etc to avoid making those scripts.
Post reply on HN