A timesyncd failure and systemd's lack of debugability • Hacker News Board

Re: A timesyncd failure and systemd's lack of debugability

#81

Post by beagle3 » Thu, Aug 09, 2018, 9:06 AM UTC

I have switched between ntpd, openntpd, timesyncd and a couple of others I might have forgotten, and the only one that's worked reliably is ntpd, but I was trying to stay off it, because historically it had quite a few vulnerabilities.

I manage hundred of stand-along edge devices, mostly hardened PCs who have to bear up to 50 deg celsius environment from time to time; I've observered 5-10 times that RTCs were wrong by 5-10 hours in some direction after a power-loss, and openntpd fails miserably, timesyncd also not-so-good. ntp does ok. It's been much better since I started updating the rtc every 10 minutes from the system clock - most distros update from rtc on startup and to rtc on shutdown, but if the shutdown is after 6 months due to power loss, you're left with an rtc drift, which -- in harsh environment -- can be significant.

I am right now testing chrony, which seems to be a winner, but it's too early to tell.

Re: A timesyncd failure and systemd's lack of debugability

#82

Post by irishsultan » Thu, Aug 09, 2018, 9:15 AM UTC

andrewstuart wrote: post #21

hedora wrote: post #17

Earlier quoted context omitted.

But you wrote: > I think "who is saying this, doing they have an underlying bias against this thing they are attacking"?

As kemitche says, you have incorrectly said that I attacked the blog post author. There are so many systemd haters out there that I immediately wonder about the motives when I read a new attack on systemd. Pointing this out is not attacking the author.

"I'm just asking the question" is hardly a valid defense against ad-hominem. If you're wondering about it and don't take the effort to research then that's your problem, not something you should share with the world. If on the other hand you have done the research, but can't be bothered to share the results the question is again why share the question with the world without sharing the answer?

Re: A timesyncd failure and systemd's lack of debugability

#83

Post by fps_doug » Thu, Aug 09, 2018, 9:19 AM UTC

majewsky wrote: post #72

iforgotpassword wrote: post #61

I'm repeating myself, but the best way to use systemd is as an init system only, and then staying away from those fancy features like dynamic users (wtf) or socket activation. Especially avoid all that bolted on crap like timesyncd, resolved, etc. Yes they integrate a bit better with systemd, but no you don't need this better integration, especially since these tools are usually less configurable than their bigger br…

> Especially avoid all that bolted on crap like timesyncd, resolved, etc. You're pretty much stuck with systemd-resolved when you use systemd-networkd. And systemd-networkd is a godsend . It's the first network setup tool that I've found that is: - full-featured - simple to set up for the 99% usecases like "just do DHCP whenever a physical interfaces goes UP" - available and works the same on all major distros - not…

Agree, this is the only thing that's almost as well-designed as the init part of systemd. Unfortunately, you still have some weird edge-case every now and then with it that's just not possible to set up, and then you're screwed because you cannot just plug in in a little shell snippet somewhere.

Re: A timesyncd failure and systemd's lack of debugability

#84

Post by beagle3 » Thu, Aug 09, 2018, 9:21 AM UTC

thwarted wrote: post #42

mikekchar wrote: post #40

Earlier quoted context omitted.

You're getting down voted mainly because you are wrong. I'm actually not well versed in this, so hopefully somebody will correct my wrong version -- but at least I think it will steer you in the right direction ;-) Systemd solves real problems that have to do with virtualisation and/or containerisation. Red Hat is the company that wrote Systemd and they did it for commercial reasons (i.e. they needed these facilities…

Systemd solves real problems that have to do with virtualisation and/or containerisation. systemd predates the industry love affair with containerization and virtualization for workload management, although work on containerization within systemd may have helped increase awareness of containerization's value. I don't remember early reasons to use systemd mentioning either; the rationales trotted out in favor of syste…

systemd does not predate the industry's love affair.

According to wikipedia, the initial release was in March 2010. The industry started courting virtualization (as it is practiced today) with the appearance of EC2, in August 2006, and the love affair came shortly afterwards.

"Reducing boot time" is of great importance for short-lived containers - if it takes 60 seconds to boot a server, and you only run it for 120 seconds, then you have 50% overhead. If it takes 6 seconds, you're down to 5% overhead; If it takes 150ms (e.g. Intel Clear Container project), it's completely negligible.

Those scripts, are, in fact, old and fragile, and while they did manage the boot process, "reliable" is a stretch. I've personally filed 2 bugs against the horrible hodge-podge of isc-dhcp that, in rare occasions, randomly failed (which is a bit of a problem if your DHCPd box is unattended and a few hours drive).

Through the years, many systems much better than init scripts have been proposed and implemented - personally I like the daemontools family, which I think is inherently better thought out.

although it definitely helped to have red hat backing, systemd won on for technical reasons: it's much better than the init scripts; it's not inferior to any other system; and it comes with simple integration with network/resolvers/etc (by virtue of including those services) -- something that saves distributions a lot of work integrating various parts.

Re: A timesyncd failure and systemd's lack of debugability

#85

Post by steevdave » Thu, Aug 09, 2018, 9:33 AM UTC

jononor wrote: post #67

steevdave wrote: post #56

Earlier quoted context omitted.

The lack of contributors has a lot to do with Lennart's (apologies if I misspelled his name) attitude early on in systemd's development. I have no idea if it has changed, as I gave up trying to get things fixed. The one issue I still have with systemd is the way it handles the kernel's command line. I work with Chromebooks a lot, using software that isn't ChromeOS. The cool thing is that they can use FIT images so yo…

Lack of contributors? systemd had over 300 different contributors the last 12 months. That makes it in the top 2% of open source projects (tracked by OpenHub). Which is very impressive for being such a low level component. https://www.openhub.net/p/systemd

Core contributors. Not overall contributors. I should have specified but figured people could read the context based on the comment I was replying to.

Re: A timesyncd failure and systemd's lack of debugability

#86

Post by bantunes » Thu, Aug 09, 2018, 9:34 AM UTC

crististm wrote: post #76

andrewstuart wrote: post #6

Before systemd, a difficult bug was a difficult bug. With systemd, a difficult bug leads to an attack on systemd itself, and questioning whether it should even exist. I like systemd, it's a pity some vocal people aren't enthusiastic about it. Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking…

Systemd is the light that people don't want to look at and embrace, heh? What I see is a an appalling amount of arrogance that comes with the "I'll solve all your current and future problems". Systemd actually doesn't and in fact can't do what it pretends. The evidence is in the never ending list of posts like these that pop up all the time. And the arrogance of pretending to be the top of the init.d hierarchy of ini…

> The evidence is in the never ending list of posts like these that pop up all the time.

So posts hating on something are proof that something doesn't work?

The open-source way used to be "come up with an alternative and win on merit". These days, since what systemd is trying to do is complex, it seems to be "complain about it online but keep using it" which leads to a super toxic user base, which might explain why bugfixing it (since you have to interact with said users) is so hard.

Re: A timesyncd failure and systemd's lack of debugability

#87

Post by bionoid » Thu, Aug 09, 2018, 9:41 AM UTC

thaumaturgy wrote: post #34

Let's carefully step around the whole distraction that is systemd. Instead: > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. This see…

> This seems to be really common practice in modern software > development, and it sucks.

I recently deployed an upgrade of an "enterprise-type" jboss application, and it failed. The log file was literally 200kb of stack traces with some "INFO" entries sprinkled here and there.

I sent it to the vendor, and their response was "everything looks okay"

Re: A timesyncd failure and systemd's lack of debugability

#88

Post by dantiberian » Thu, Aug 09, 2018, 9:43 AM UTC

gnud wrote: post #75

andrewstuart wrote: post #6

Before systemd, a difficult bug was a difficult bug. With systemd, a difficult bug leads to an attack on systemd itself, and questioning whether it should even exist. I like systemd, it's a pity some vocal people aren't enthusiastic about it. Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking…

You obviously didn't read the article before deciding this wasn't systemd's fault. The bug is the lack of any error handling in the DynamicUser feature of systemd. The original failure is not logged anywhere - and launching timesyncd just goes ahead even though the setup for it fails completely. Naturally timesyncd then fails to start. (Or perhaps the bug is the whole DynamicUser feature. From reading the bug report,…

The other part necessary to trigger the bug is that uid 0 is not root and is not able to read the user-only mounts. root would still be able to read the user-only FUSE mount.

Re: A timesyncd failure and systemd's lack of debugability

#89

Post by wokwokwok » Thu, Aug 09, 2018, 9:45 AM UTC

lolc wrote: post #47

wokwokwok wrote: post #16

Earlier quoted context omitted.

spins finger in the air the difference is immaterial and irrelevant in this context.

A huge part of GNU/Linux is written in C. A small part is written in C++. I would accept the argument that C++ is less accessible than C to people maintaining Linux systems. Based on this alone I would say that yes, the difference is material and relevant.

pish posh. It could be written in Egyptian and it would make no material difference to the ease of debugging for most people.

The number of people who are C++ programmers and can deal with strace vs. C programmers who can deal with strace is really that significant? I don't buy it. The two groups intersect significantly.

The point is that other people who are not C or C++ programmers have to deal with this kind of stuff from systemd.

This is just pedantry for the sake of it.

Systemd would be only marginally harder to debug if it was in C++ or heck, even rust, or go.

No one is opening the source code here.

Re: A timesyncd failure and systemd's lack of debugability

#90

Post by teddyh » Thu, Aug 09, 2018, 9:50 AM UTC

steevdave wrote: post #56

newnewpdro wrote: post #28

TL;DR: systemd has bugs. Mountain out of a mole hill. Lack of debuggability? It's not like this is some threaded golang program you'll lose your mind attempting to make sense of in strace. systemd is a single-threaded C program, if you consider this as not debuggable it's your failing not systemd's. There are valid complaints to be made about systemd, but from what I've seen the crux of the problem is generally the h…

The lack of contributors has a lot to do with Lennart's (apologies if I misspelled his name) attitude early on in systemd's development. I have no idea if it has changed, as I gave up trying to get things fixed. The one issue I still have with systemd is the way it handles the kernel's command line. I work with Chromebooks a lot, using software that isn't ChromeOS. The cool thing is that they can use FIT images so yo…

> adds a flag to the kernel command line "debug" - systemd sees that and thinks you want systemd in debug mode

A reasonable assumption, given that systemd is taking care of the whole OS, and if you want the kernel in debug mode, you’d reasonably want the system in debug mode too. But it’s not obviously correct, either, and ultimately it boils down to “who owns the kernel command line flags”. There was some kerfuffle about exactly that between the kernel and systemd people, but they eventually worked it out – I think systemd no longer tries to assert ownership of the "debug" kernel command line parameter.