Live data from Hacker News

In search of a better job scheduler

beepb00p.xyz

21–30 of 30 posts

Re: In search of a better job scheduler

#21

I love all the options we have today. At Cronitor we are seeing a lot of users migrating to Airflow, Kubernetes cron jobs, and scheduled lambdas. Like databases, different workloads will benefit from different guarantees. Look for the right tool for the job, not the right tool for all time.

As someone migrating from airflow back to cron jobs (I can do that doesn't really need to be distributed), I can't agree more. And the important thing is to look at the complexity of the solution as well.

Re: In search of a better job scheduler

#22

Is there a reason not to consider apache airflow? https://airflow.apache.org/ You could run it on a digital ocean droplet so you don't have to worry about your laptop turning off. It covers retry and cron-style job kickoffs.

Complexity, effectively not supporting shorter scheduling interval (if you need to run something more frequently than every 20 minutes, beware), some bugs don't get fixed for years and are very hard to debug. We ended up restating the scheduler on regular Cron to kick stale task queue.

Re: In search of a better job scheduler

#23

You can create systemd timers and services quite easily with NixOS configuration : https://nixos.wiki/wiki/Nix_Cookbook

The issue AFAICT is that you need to define a separate unit to run the command. It's not clear why systemd has this restriction on timers, other than implementation convenience.

Re: In search of a better job scheduler

#24
post #9

Jenkins works reasonably well as a job scheduler. The UI is functional if not pretty, it is fairly easy to configure and use even for a large number of jobs, and the plugin ecosystem allows you to extend it in many ways without writing any code.

It would be kinda nice if we could pull the job runner out as it’s own thing.

Re: In search of a better job scheduler

#25
Slightly tangential but all I could think about while reading this was: Ansible. I switched to it about 6 months back - including for managing e.g. cron jobs and systemd - and I can’t imagine going back to doing any of this by hand. High-level abstraction and jinja2 templating makes this stuff a breeze.

Re: In search of a better job scheduler

#26
post #12

I don't get the systemd hate in this specific case. Their timers are pretty much the best possible option you have to schedule jobs, hands down. No cron Daemon is ever going to get there due to the lack of deep integration with the rest of the init system. I know it takes a while to get used to the ini file madness, but honestly, I can't detect so much boilerplate there: every single line in my units and timers makes…

Also `systemctl edit` is a thing. No need to fiddle with 'what directory is this thing in?'

Re: In search of a better job scheduler

#27
I automated the tedious work to create systemd timers in an ansible role: https://yourlabs.io/oss/yourlabs.timer

As a result, I can execute it include it in another ansible role, or use bigsudo to apply it as a one-off command:

    bigsudo yourlabs.timer @somehost name=your-backup cmd=/your/backup.sh oncalendar='*-*-* 00:00:00 Europe/Paris'

Re: In search of a better job scheduler

#28
post #12

I don't get the systemd hate in this specific case. Their timers are pretty much the best possible option you have to schedule jobs, hands down. No cron Daemon is ever going to get there due to the lack of deep integration with the rest of the init system. I know it takes a while to get used to the ini file madness, but honestly, I can't detect so much boilerplate there: every single line in my units and timers makes…

Also `systemctl edit` is a thing. No need to fiddle with 'what directory is this thing in?'

Yeah with the --full option you can be sure you've got your own full copy to work with in /etc

Re: In search of a better job scheduler

#29

Earlier quoted context omitted.

Also `systemctl edit` is a thing. No need to fiddle with 'what directory is this thing in?'

Yeah with the --full option you can be sure you've got your own full copy to work with in /etc

Huh, I didn't know about --full. Thanks, it does help with some of this pain.
Post reply on HN