Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A timesyncd failure and systemd's lack of debugability

utcc.utoronto.ca

Post reply on HN
Print view View original

111–120 of 126 posts

Re: A timesyncd failure and systemd's lack of debugability

#111

Post by beagle3 »

rkeene2 wrote: post #105
beagle3 wrote: post #81

I have switched between ntpd, openntpd, timesyncd and a couple of others I might have forgotten, and the only one that's worked reliably is ntpd, but I was trying to stay off it, because historically it had quite a few vulnerabilities. I manage hundred of stand-along edge devices, mostly hardened PCs who have to bear up to 50 deg celsius environment from time to time; I've observered 5-10 times that RTCs were wrong b…

You could always use htpd [0]. :-) [0] http://rkeene.org/docs/oss/htp/htp.pdf

For various reasons, the powers that be mandated ntp; but thanks for the reference.

Re: A timesyncd failure and systemd's lack of debugability

#113

Post by dantiberian »

kps wrote: post #104
dantiberian wrote: post #88

Earlier quoted context omitted.

The other part necessary to trigger the bug is that uid 0 is not root and is not able to read the user-only mounts. root would still be able to read the user-only FUSE mount.

User 0 is root, and it is not able to read everything on other machines — as has been the case for decades.

Perhaps I misunderstood this line then?

> The machine that this happened on is an NFS client and (as is usual) its UID 0 is mapped to an unprivileged UID on our fileservers. On this machine there were some FUSE mounts in the home directories of users who have their $HOME not world readable (our default $HOME permissions are owner-only, to avoid accidents). When systemd was setting up the 'slightly modified mount name-space' it attempted to access these FUSE mounts as part of binding them into the namespace, but it failed because UID 0 had no permissions to look inside user home directories.

https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynami...

Re: A timesyncd failure and systemd's lack of debugability

#114

Post by kps »

dantiberian wrote: post #113
kps wrote: post #104

Earlier quoted context omitted.

User 0 is root, and it is not able to read everything on other machines — as has been the case for decades.

Perhaps I misunderstood this line then? > The machine that this happened on is an NFS client and (as is usual) its UID 0 is mapped to an unprivileged UID on our fileservers. On this machine there were some FUSE mounts in the home directories of users who have their $HOME not world readable (our default $HOME permissions are owner-only, to avoid accidents). When systemd was setting up the 'slightly modified mount name…

My reading is that (a) systemd can't read home directories on the file server, because clients understandably don't get root permissions there; and (b) the reason systemd was trying to look in those directories is that they contained (non-root-users') FUSE mount points.

Re: A timesyncd failure and systemd's lack of debugability

#115

Post by muxator »

Teknoman117 wrote: post #29
dijit wrote: post #26

Earlier quoted context omitted.

False dichotomy. I pay redhat a lot of money, short of paying people full time to maintain a mirror distro of redhat without systemd, which would be impossible given how embedded it has become into everything. Some decisions cannot be unmade. Short of starting from scratch with gentoo (which is a Herculean effort at scale) I’m not sure what you’re saying. I cannot be disgruntled at choices because distro maintainers…

The company I work for runs Gentoo at a global scale...

You live in a better place!

Re: A timesyncd failure and systemd's lack of debugability

#116

Post by muxator »

iforgotpassword wrote: post #61

I'm repeating myself, but the best way to use systemd is as an init system only, and then staying away from those fancy features like dynamic users (wtf) or socket activation. Especially avoid all that bolted on crap like timesyncd, resolved, etc. Yes they integrate a bit better with systemd, but no you don't need this better integration, especially since these tools are usually less configurable than their bigger br…

In that case Ubuntu was a middleman whose interests were completely disaligned with their customers', which probably means they in reality felt obliged to someone else.

You tried to voice. Depending on your loyalty, you may think of exiting [0].

[0] https://en.wikipedia.org/wiki/Exit,_Voice,_and_Loyalty

Re: A timesyncd failure and systemd's lack of debugability

#117

Post by newnewpdro »

jononor wrote: post #67
steevdave wrote: post #56

Earlier quoted context omitted.

The lack of contributors has a lot to do with Lennart's (apologies if I misspelled his name) attitude early on in systemd's development. I have no idea if it has changed, as I gave up trying to get things fixed. The one issue I still have with systemd is the way it handles the kernel's command line. I work with Chromebooks a lot, using software that isn't ChromeOS. The cool thing is that they can use FIT images so yo…

Lack of contributors? systemd had over 300 different contributors the last 12 months. That makes it in the top 2% of open source projects (tracked by OpenHub). Which is very impressive for being such a low level component. https://www.openhub.net/p/systemd

Anyone who follows the systemd-devel mailing list and the systemd github repo knows that there's just around four maintainers regularly active to varying degrees, including Lennart.

Re: A timesyncd failure and systemd's lack of debugability

#118

Post by Terr_ »

hvidgaard wrote: post #73
thaumaturgy wrote: post #34

Let's carefully step around the whole distraction that is systemd. Instead: > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. This see…

> > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. > This seems to be really common practice in modern software development, and it s…

> It is impossible to give meaningful error messages for all errors.

Not for end-users, no, because you're faced with a lot of future unknowns about who they are and what they were thinking of doing.

For developers, on the other hand... IMO this has similarities to the "exceptions or return-type checking" debate. Certain architectural choices will have a strong impact on whether the inevitable runtime/production failures are mysterious or not.

Re: A timesyncd failure and systemd's lack of debugability

#119

Post by josteink »

dijit wrote: post #26
cwyers wrote: post #25

Earlier quoted context omitted.

Distro maintainers didn't force you to do anything. They made a decision about how to allocate their limited resources. And we're past the days of Linux being a primarily volunteer OS, but a fair proportion of distro maintainers are still volunteers. They are not at your beck and call. If you want to have a Linux distro without systemd, there's several. If none of them meet your needs, pitch in with time or money to…

False dichotomy. I pay redhat a lot of money, short of paying people full time to maintain a mirror distro of redhat without systemd, which would be impossible given how embedded it has become into everything. Some decisions cannot be unmade. Short of starting from scratch with gentoo (which is a Herculean effort at scale) I’m not sure what you’re saying. I cannot be disgruntled at choices because distro maintainers…

> False dichotomy. I pay redhat a lot of money

Well why are you paying for a distro which doesn't meet your wishes then? Why not choose another one?

Re: A timesyncd failure and systemd's lack of debugability

#120

Post by KaiserPro »

zaarn wrote: post #66
KaiserPro wrote: post #37

Earlier quoted context omitted.

Before systemd the error were shell errors. before systemd the bundled logger wrote to a file, didn't direct people to journald, that lags like a champ when a log file is > 1 meg. before systemd errors in config files were on specific lines I don't have bias against systemd I have bias against it's flaws. I like its config file. I hate it's documentation. I like that it has a cron like feature, I hate that its pretty…

sysvinit has it's warts too and it can get quite ugly. For example, atm I need to have 3 services. Two of these services need to run at the same time, they can run separately but then they are not useful. The third services can only run with both services online and is first run 15 minutes after boot (not earlier!) and then every hour but the hourly run MUST NOT run before the first run of the boot (ie the very first…

I used supervisord for anything complex, which is where I think systemd get's it's syntax from.

Anisble abstracts some stuff (ensure running, running as use, etc) but for more complex stuff (process type, dependency management etc) you're on your own shipping system files.

My main beef is actually three things:

1) crap error messages,

2) crap validataion

3) journalctl being a crap VIM wrapper.

of all of them, journalctl is the most obnoxious. Instead of deffering to the system viewer it enforces their crap viewer. It truncates by default, and without restarting the viewer, or forcing it through less. (Why didn't they use less by default? less code to maintain..)

Post reply on HN