In search of a better job scheduler
beepb00p.xyz
In search of a better job scheduler
1–10 of 30 posts
Re: In search of a better job scheduler
#2One more:
Re: In search of a better job scheduler
#3I'm pretty sure something like Kube could be used to do this using CronJobs, pod affinities, and Prometheus+AlertManager for alerts and notifications.
You can specify memory, CPU, retries, monitor, and run on many computers.
I think Kube is like systemd + cron + containers had a baby.
Re: In search of a better job scheduler
#4I know this is mac specific, but since op is looking for personal infra level tooling/scheduling, there might be an equivalent what ical offers:
create ical event+repeat+alert=Custom(Open file)
this will open a file every repeat event. The file can be shell/workflow or whatever you want to run.
Re: In search of a better job scheduler
#5«
better specs for jobs
dependencies, timeouts, resource policies and retries without hacky wrappers and boilerplate »
This is what I miss.
There's a strange gap in traditional Unix there, compared to other time-sharing systems. I have a feeling that if Unix had had a primitive job manager then by 1990 there would have been a pile of Gnu extensions and by 2000 there would have been people reinventing it, and GUIs that looked a bit like what you get to monitor print queues.
Re: In search of a better job scheduler
#6One more: Rundeck https://docs.rundeck.com/docs/manual/
Re: In search of a better job scheduler
#7I work on a meta scheduler that solves the problem of periodicity on cron - https://cylc.github.io/
It was created to manage weather forecast model runs. The new version which we are working on will have a simpler web GUI to replace its old PyGTK and Py2.
It handles o ly scheduling, and supports cyclic graphs (contrary to most workflow managers build with DAGs only).
No Python API yet, but coming in future release. So only a custom suite.rc INI style file that supports Jinja2 too.
Re: In search of a better job scheduler
#8Anyone have a good open source replacement for Autosys or controlM? Should be easy to replace but every enterprise I've seen still pays for one of those.
Re: In search of a better job scheduler
#9Jenkins works reasonably well as a job scheduler. The UI is functional if not pretty, it is fairly easy to configure and use even for a large number of jobs, and the plugin ecosystem allows you to extend it in many ways without writing any code.
Re: In search of a better job scheduler
#10Jenkins works reasonably well as a job scheduler. The UI is functional if not pretty, it is fairly easy to configure and use even for a large number of jobs, and the plugin ecosystem allows you to extend it in many ways without writing any code.
Any decent CI server should do, though exponential back off isn’t something that I’m aware can be configured without custom scripting.