Live data from Hacker News

Advent of Sysadmin 2025

sadservers.com

131–140 of 140 posts

Re: Advent of Sysadmin 2025

#131
Maybe I'm just extremely dumb, but I can't find how to edit files? Neither `vi` nor `nano` are installed, I don't have internet access to `apt-get update`, and I'm not about to learn `emacs` for this...

EDIT: Ah, ok, `vi` is installed on the server _itself_, just not in the Docker containers. So I guess I'm going to have to `docker cp` them in. Can do o7

Re: Advent of Sysadmin 2025

#132
post #109

Earlier quoted context omitted.

I really scratch my head when I read your comment, as nothing of this is a real issue in my Jenkins. > bunch of random Jenkins servers Either PXE boot from an image, or k8s from an image, have a machine or pod rebooted/destroyed after one job. Update your image once a month, or have a Jenkins job to do that for you. > Authentication, authorization, access control Either use LDAP or Login via Github, and Matrix securi…

Well one issue is, CasC isn't enough. You often have to write JobDSL to get around some limitation in CasC, and sometimes Groovy for limitations in the other two. If you want to manage access control (and you choose the correct Auth plugin, and figure out how to configure it), often you need an admin to make changes in both the Jenkins server and your backend AuthNZ system. Then there's the "seed job vs not-seed-job"…

My issue with GHA and other "dead simple" systems is that my CI is complicated. Having a real programming language for stuff like "calculate what date it was a week ago" or "concatenate these three strings but only under some conditions" or "parse the output and build an object out of it" is really helpful while a bastardised YAML-based Jinja template simply can't hold up.

But yeah, if all there is to do is "git clone && build && deploy" then Jenkins is an overkill and it probably wasn't warranted in the first place.

Re: Advent of Sysadmin 2025

#133
post #123

Cool, might try it out! Are there any solutions repositories for them. I’d love to get an explanation for the ones I’m about to fail.

Personal advice: don't use solutions repo. Googling the problem and then digging deep into the solutions will teach you hell lot more. Read the man pages of commands that turn up on Google, try them with different options, try to find different commands which can do almost the same thing may be a bit differently .... all these will help you learn things lot more.

Re: Advent of Sysadmin 2025

#134
post #132

Earlier quoted context omitted.

Well one issue is, CasC isn't enough. You often have to write JobDSL to get around some limitation in CasC, and sometimes Groovy for limitations in the other two. If you want to manage access control (and you choose the correct Auth plugin, and figure out how to configure it), often you need an admin to make changes in both the Jenkins server and your backend AuthNZ system. Then there's the "seed job vs not-seed-job"…

My issue with GHA and other "dead simple" systems is that my CI is complicated. Having a real programming language for stuff like "calculate what date it was a week ago" or "concatenate these three strings but only under some conditions" or "parse the output and build an object out of it" is really helpful while a bastardised YAML-based Jinja template simply can't hold up. But yeah, if all there is to do is "git clon…

For complex logic I don't rely on the CI system; I've been burned too many times. I shell out to an external program and have it return an output variable, and I just do "if $foo = y then blah" in the CI's DSL (and I keep those tests to a minimum; rather have more separate jobs than one complex job). Often I will put everything in a dedicated build tool (Make or similar) so I can run it from my laptop or CI, and any change to logic only happens in one place. It's adding an abstraction, but the end result is I write the CI job once and never touch it again. For flexibility I add parameters to the CI job.

Re: Advent of Sysadmin 2025

#135
post #121

Earlier quoted context omitted.

Not in love with its insistence on recreating the container from scratch every step of the pipeline, among a bundle of other irksome quirks. There are certainly worse choices, though.

You don't actually have to. If you use the built in container registry and build artifacts, you can pass between steps.

I'm aware, but thank you. Unfortunately, given sufficiently large artifacts, the overhead of packaging, uploading, downloading and unpacking them at every step becomes prohibitive.

Re: Advent of Sysadmin 2025

#136

Earlier quoted context omitted.

This is documented currently (supposed to be 14 days). [1] That said, I have found runners to be unnecessarily difficult. But Jenkins and its own quirks, and when I used GitLab, it used ancient docker-machine and outdated AMIs by default. I think Buildkite has been the only one to make this easy and scalable. But it is meant for self hosted runners. [1] https://docs.github.com/en/enterprise-cloud@latest/actions/h...

Buildkite also has hosted runners (which they all agents): https://buildkite.com/docs/pipelines/hosted-agents

It does, but that came second.

It was originally (and still usually) used by those who wanted to self-host runners.

Re: Advent of Sysadmin 2025

#137

Here's 12 Sysadmin/DevOps (they're synonyms now!) challenges, straight from the day job: 1. Get a user to stop logging in as root. 2. Get all users to stop sharing the same login and password for all servers. 3. Get a user to upgrade their app's dependencies to versions newer than 2010. 4. Get a user to use configuration management rather than scp'ing config files from their laptop to the server. 5. Get a user to bak…

I know its a common view that sysadmin/devops are the same these days, but witha current sysadmin role nothing youve mentioned sounds relevant. Let's give you my list: 1. Patch Microsoft exchange with only a three hour outage window 2. Train a user to use onedrive instead of emailing 50mb files and back and forth 3. Setup eight printers for six users. Deal with 9gb printer drivers. 4. Ask an exec if he would please l…

>4. Ask an exec if he would please let you add mfa to their mailbox.

Ask?! This is where the org's cyber insurance is your friend. Just have the executive get the provider's clearance on him not having MFA. I'm sure that line item will change his mind, and if not, be sure to accidently mention those exemptions to those yearly auditors.

Re: Advent of Sysadmin 2025

#138
post #132

Earlier quoted context omitted.

My issue with GHA and other "dead simple" systems is that my CI is complicated. Having a real programming language for stuff like "calculate what date it was a week ago" or "concatenate these three strings but only under some conditions" or "parse the output and build an object out of it" is really helpful while a bastardised YAML-based Jinja template simply can't hold up. But yeah, if all there is to do is "git clon…

For complex logic I don't rely on the CI system; I've been burned too many times. I shell out to an external program and have it return an output variable, and I just do "if $foo = y then blah" in the CI's DSL (and I keep those tests to a minimum; rather have more separate jobs than one complex job). Often I will put everything in a dedicated build tool (Make or similar) so I can run it from my laptop or CI, and any…

> For complex logic I don't rely on the CI system

I do that too, until my complex logic belongs to the CI system and nowhere else.

As an example:

    It's 2 am now and Jenkins needs to run some (but not all) nightly tests[0]. To figure out which, let's bring the source code and analyse the configuration file, disregarding anything that doesn't need to run at this hour. Once we have the plan for which tests to run, let's figure out what we need to build[1]. Also, let's see what is the status of the pool running the tests[2] so we can decide on a tests parallelisation strategy for this run. When we have a plan, let's build and test, keeping an eye on the triggered tests. When all these have finished, analyse the failures[3], create tickets for the failures[4] and prepare a report to be sent.
I wouldn't be able to express all this in YAML.

[0] other "nightly tests" run at 1, 3, 4 etc.

[1] this is mapped in the configuration file too.

[2] this is internal to Jenkins

[3] same

[4] this involves finding the "responsible person," so a lot of API calls

Re: Advent of Sysadmin 2025

#139
post #138

Earlier quoted context omitted.

For complex logic I don't rely on the CI system; I've been burned too many times. I shell out to an external program and have it return an output variable, and I just do "if $foo = y then blah" in the CI's DSL (and I keep those tests to a minimum; rather have more separate jobs than one complex job). Often I will put everything in a dedicated build tool (Make or similar) so I can run it from my laptop or CI, and any…

> For complex logic I don't rely on the CI system I do that too, until my complex logic belongs to the CI system and nowhere else. As an example: It's 2 am now and Jenkins needs to run some (but not all) nightly tests[0]. To figure out which, let's bring the source code and analyse the configuration file, disregarding anything that doesn't need to run at this hour. Once we have the plan for which tests to run, let's…

Sounds like you're using cron as a complex job queue! A lot of teams get there eventually, and either 1) keep hacking on cron/jenkins/etc to make this work, 2) invent their own queueing tool (NIH syndrome; been done many times before, there is nothing new to make here), or 3) use a purpose-built solution for this. Airflow is the old-and-busted solution; the new hotness is newer generations of the same concept (Prefect/Dagster, Luigi, Temporal). But often sticking to your existing thing is cheaper; depends how much custom engineering you want to invest.

Fwiw, I do believe you can do this in GHA, but you may need to call their API from your workflow. In addition, their replacement for Groovy is to run an action which lets you embed Javascript/Typescript and call their SDK. It sucks, but so does Groovy! ;-)

Re: Advent of Sysadmin 2025

#140
post #24
post #2

[flagged]

Checking out how the platform works was two clicks away: home -> give me a server. I don't know of any other SaaS which gives you a VM with one click without any registration but we do it. In any case thanks for the feedback, I've put a button on this /advent page for clarity, cheers

This text:

> Sign up for a free account (needed to keep track of your progress)

is a complete lie. Tracking a person’s progress is what cookies are for. You don’t need us to create an account for that.

What you do need users to create accounts for, is for you to track every user and their progress.

Post reply on HN