A timesyncd failure and systemd's lack of debugability
utcc.utoronto.ca
A timesyncd failure and systemd's lack of debugability
1–10 of 126 posts
Re: A timesyncd failure and systemd's lack of debugability
#2Given things like this: "The default configuration is defined during compilation" [1] and the fact that you'd think 'timedatectl status' would tell you the NTP server you're supposedly trying to sync to (spoiler alert: it doesn't), I'm unsurprised of any further atrocities committed by systemd.
[1]: https://www.freedesktop.org/software/systemd/man/timesyncd.c...
Re: A timesyncd failure and systemd's lack of debugability
#3The follow up article (https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynami...) explains the underlying problem.
Basically, to get around root-only readable directories, a special mount namespace is created when Dynamic Users are used. That fails when there are mount points that systemd can't access, which was due to FUSE filesystems mounted inside owner-only home directories, combined with an NFS mount.
This is the second time in a few days that I've heard about systemd-related user / ownership problems. The other was related to when a systemd-managed socket needs to be owned by a user that is created by a daemon that relies on networking to already be up (ldap / activedirectory / whatever). systemd wants to create the socket at the same time as networking is starting, so the user doesn't exist yet -- and there isn't a simple way to tell systemd to delay creating that socket.
Instead, the post-start script of the user daemon can chown the socket, and services relying on it will need to be started after that -- and there is no generic way to say to start a service after the auth provider is up, so you have to be specific, making the dependency fragile.
Re: A timesyncd failure and systemd's lack of debugability
#4It might be config and security fragile, but ntpd is a safe bet.
When religions preferences come up, I prefer s6 because it’s scalable like daemon tools/runit. Old-school upstart was meh and ancient style rc scripts rarely did process supervision.
Debugging and simplicity are prime requirements for init’s.
http://www.skarnet.org/software/s6/
PS: Logs aren’t files... they’re streams of structured messages often, that are unfortunately usually destructured into text lines. Log-rotation, logging to local files are horrible kludges that doesn’t scale for high-volume servers... log-structured log device, log sourcing or time-series structured db are more natural fits. Keeping log data in structured form instead of broken up in arbitrary line formats makes custom parsing unnecessary and log mining/transformation (ETL) much easier and usable. “Grepping” is simply looking at a text view of desired fields like a full-text db search. IOW, historical logs become random-access msgpack/protobufs streams while app logs are just always appending... and one string per log message for backwards compat, with a “logger”-type command for logfile-less 12-factor apps.
Re: A timesyncd failure and systemd's lack of debugability
#5systemd's job is to start daemons. It can be reasonable to decide dynamically when to start them based on which others are healthy (maybe you care more about speed than determinism on a laptop). But there's no reason to choose a uid and mkdir dynamically. You want these errors at install time, not at the box's next (probably unattended) restart.
Re: A timesyncd failure and systemd's lack of debugability
#6Before systemd, a difficult bug was a difficult bug.
With systemd, a difficult bug leads to an attack on systemd itself, and questioning whether it should even exist.
I like systemd, it's a pity some vocal people aren't enthusiastic about it.
Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking"?
I guess my question would be, is this really a general issue about systemd being difficult to debug, or some specific component subsystem of systemd. systemd is a big place.
Re: A timesyncd failure and systemd's lack of debugability
#7I'm always amazed at what systemd can do that I don't know about. I had no idea about timesyncd. How did anyone find out about it? Is there a super awesome book or video of systemd "hacks" that I can read or watch? I think I've read the official docs but I don't think I really learned anything, but that was a couple of years ago.
Edit: Wow, just realized that my debian laptop is running timesyncd. I run the same debian version on all my servers too but I installed ntpd on them. I guess I need to check how their conflicting, if any...
Re: A timesyncd failure and systemd's lack of debugability
#8> this server drifted more than half a second out of synchronization before I found this issue entirely by coincidence
Did `systemctl status` not report "degraded" for this failure? Maybe the problem is that there was no monitoring set up for failing services?
Re: A timesyncd failure and systemd's lack of debugability
#9Given things like this: "The default configuration is defined during compilation" [1] and the fact that you'd think 'timedatectl status' would tell you the NTP server you're supposedly trying to sync to (spoiler alert: it doesn't), I'm unsurprised of any further atrocities committed by systemd. [1]: https://www.freedesktop.org/software/systemd/man/timesyncd.c...
> Given things like this: "The default configuration is defined during compilation [so a configuration file is only needed when it is necessary to deviate from those defaults....]"
I don't get it. What's so bad about that?
Re: A timesyncd failure and systemd's lack of debugability
#10I'm always amazed at what systemd can do that I don't know about. I had no idea about timesyncd. How did anyone find out about it? Is there a super awesome book or video of systemd "hacks" that I can read or watch? I think I've read the official docs but I don't think I really learned anything, but that was a couple of years ago. Edit: Wow, just realized that my debian laptop is running timesyncd. I run the same debi…
as a start, try "man -k systemd"