Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A timesyncd failure and systemd's lack of debugability

utcc.utoronto.ca

Post reply on HN
Print view View original

121–126 of 126 posts

Re: A timesyncd failure and systemd's lack of debugability

#121

Post by zaarn »

KaiserPro wrote: post #120
zaarn wrote: post #66

Earlier quoted context omitted.

sysvinit has it's warts too and it can get quite ugly. For example, atm I need to have 3 services. Two of these services need to run at the same time, they can run separately but then they are not useful. The third services can only run with both services online and is first run 15 minutes after boot (not earlier!) and then every hour but the hourly run MUST NOT run before the first run of the boot (ie the very first…

I used supervisord for anything complex, which is where I think systemd get's it's syntax from. Anisble abstracts some stuff (ensure running, running as use, etc) but for more complex stuff (process type, dependency management etc) you're on your own shipping system files. My main beef is actually three things: 1) crap error messages, 2) crap validataion 3) journalctl being a crap VIM wrapper. of all of them, journal…

Journalctl uses less as a pager by default, you can set the env variable SYSTEMD_PAGER to specifiy what pager to use (or just PAGER) and SYSTEMD_LESS to specify parameters to pass to less.

journalctl has no own pager implementation and if no pager is installed then none is used.

I give you validation and error messages, systemd is a bit sparse on that front. Would be easier if there was a way for programs and services to signal back error messages to report.

edit:

You can actually enforce non-wrapping by setting SYSTEMD_LESS to FRXMK (the S option that is otherwise included causes truncating, if omitted you get wrapping)

This can all be configured in /etc/profile, .bashrc and friends

Re: A timesyncd failure and systemd's lack of debugability

#122

Post by amaccuish »

beagle3 wrote: post #84
thwarted wrote: post #42

Earlier quoted context omitted.

Systemd solves real problems that have to do with virtualisation and/or containerisation. systemd predates the industry love affair with containerization and virtualization for workload management, although work on containerization within systemd may have helped increase awareness of containerization's value. I don't remember early reasons to use systemd mentioning either; the rationales trotted out in favor of syste…

systemd does not predate the industry's love affair. According to wikipedia, the initial release was in March 2010. The industry started courting virtualization (as it is practiced today) with the appearance of EC2, in August 2006, and the love affair came shortly afterwards. "Reducing boot time" is of great importance for short-lived containers - if it takes 60 seconds to boot a server, and you only run it for 120 s…

> "Reducing boot time" is of great importance for short-lived containers

Containers don't tend to run an init.

Re: A timesyncd failure and systemd's lack of debugability

#123

Post by nomel »

AnIdiotOnTheNet wrote: post #93
hvidgaard wrote: post #73

Earlier quoted context omitted.

> > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. > This seems to be really common practice in modern software development, and it s…

If you structure your software simply and make it easy to reason about what is going on, then any half decent error message should give enough information to at least start down the right path. But we don't get that in modern software. We get "Something went wrong :(".

> If you structure your software simply and make it easy to reason about what is going on

Complex problems are not easily made simple or easily reasoned about. Something like systemd (or any non trivial piece of linux) could never be made simple or, with your hands down in the guts of it, easily understood.

Breaking systemd's functionality into a million separate pieces wouldn't make the complexity go away, it would necessarily increase it.

I don't know what the solution is, but "making things simpler and easier to understand" is a bit hollow. Its problem space was never simple or easy to understand (unless you think systemd just runs some things at startup).

Re: A timesyncd failure and systemd's lack of debugability

#124

Post by digi_owl »

amaccuish wrote: post #122
beagle3 wrote: post #84

Earlier quoted context omitted.

systemd does not predate the industry's love affair. According to wikipedia, the initial release was in March 2010. The industry started courting virtualization (as it is practiced today) with the appearance of EC2, in August 2006, and the love affair came shortly afterwards. "Reducing boot time" is of great importance for short-lived containers - if it takes 60 seconds to boot a server, and you only run it for 120 s…

> "Reducing boot time" is of great importance for short-lived containers Containers don't tend to run an init.

While true, i think the notion here is to treat containers like VMs in the cloud. Thus a container is spun up or down depending on the load on the service.

Then again, that notion brings to mind a certain comic about a self-ddos...

http://www.commitstrip.com/en/2015/07/08/true-story-fixing-a...

Re: A timesyncd failure and systemd's lack of debugability

#125

Post by digi_owl »

nomel wrote: post #123
AnIdiotOnTheNet wrote: post #93

Earlier quoted context omitted.

If you structure your software simply and make it easy to reason about what is going on, then any half decent error message should give enough information to at least start down the right path. But we don't get that in modern software. We get "Something went wrong :(".

> If you structure your software simply and make it easy to reason about what is going on Complex problems are not easily made simple or easily reasoned about. Something like systemd (or any non trivial piece of linux) could never be made simple or, with your hands down in the guts of it, easily understood. Breaking systemd's functionality into a million separate pieces wouldn't make the complexity go away, it would…

The trick is to have a environment where you can take the complex apart and examine each part in isolation, and then bring them together until you replicate the error.

With older _nix setups that was fully possible because daemons were pretty much (i am likely to get dogpiled on this) a user process without a shell to send output to.

With systemd on the other hand the state inside a systemd-managed unit, and the state outside of said "black box" may be wildly different. And to debug the person doing so have to attempt to maintain a massive mental image of state.

Frankly the kernel may well be the closest approximation, and it may well be better instrumented at this point. Never mind that building a second kernel in userspace is not a kudos point.

Re: A timesyncd failure and systemd's lack of debugability

#126

Post by lomnakkus »

fowl2 wrote: post #78
lomnakkus wrote: post #58

Earlier quoted context omitted.

Personally, I think even having to choose a UID at install time or dynamically is a pretty sad indictment of just how broken the user system on Linux (and pretty much every UNIX-like) is. It leads to a huge amount of incidental complexity where there's a much simpler way: User IDs should just be randomly chosen UUIDs -- either generated by upstreams or by the distros. Any metadata attached to the user UID could just…

Except of course if you want multiple instances of the same daemon

Um, mix in a second randomly-chosen UUID?

My point wasn't so much that I had the perfect solution and that I'd thought of everything. It's that there are much better systems and that we should actually strive to get there.

Post reply on HN