Ask HN: What job scheduler do you use?

#1

Post by pmchorus » Sat, Apr 08, 2017, 8:01 AM UTC

In a previous company, I used the in-house job scheduler. Despite some bugs and lack of documentation it did what we asked for: schedule jobs according schedules and timezones, and handle dependencies.

Now I'm working for another company, we have the same need:

- Orchestrate hundreds of jobs

- Some need to start as soon as their dependencies are ready

- Others need to start according to timezone-based (and possibly dependencies)

- Notification on failures

- Make jobs fail if their dependencies are not ok after a given time

- Force a job status (i.e. set it "ok" even if it failed)

- Reasonable access control

- And if possible a nice dashboard!

While these requirements seem standard to me, I cannot find any acceptable solution:

- Cron is cool for the simplest things, nothing more

- Rundeck is nice for simple workflows, but not for complex dependencies

- I'm tinkering with SOS-Berlin Job Scheduler for days now, and I cannot find proper ways to orchestrate my dependencies

- Control-M looks over priced and certainly not deployable without months of project planing and contract negotiation

- Luidgi and Airflow are too much focused on data workflows, not on generic job scheduling

What are you using? :)

Re: Ask HN: What job scheduler do you use?

#2

Post by paktek123 » Sat, Apr 08, 2017, 9:10 AM UTC

https://mesos.github.io/chronos/

Re: Ask HN: What job scheduler do you use?

#3

Post by Xeago » Sat, Apr 08, 2017, 9:50 AM UTC

Given that you mention Luigi, have a look at Styx. Currently in use at Spotify as a complement to the underlying job processing frameworks.

https://github.com/spotify/styx

Re: Ask HN: What job scheduler do you use?

#4

Post by jacksonnic » Sat, Apr 08, 2017, 10:14 AM UTC

https://www.nomadproject.io is awesome for scheduling both long running processes and time triggered jobs

Re: Ask HN: What job scheduler do you use?

#5

Post by pmchorus » Sat, Apr 08, 2017, 11:30 AM UTC

jacksonnic wrote: post #4

https://www.nomadproject.io is awesome for scheduling both long running processes and time triggered jobs

It does not seem to handle dependencies.

Re: Ask HN: What job scheduler do you use?

#6

Post by pmchorus » Sat, Apr 08, 2017, 11:32 AM UTC

Xeago wrote: post #3

Given that you mention Luigi, have a look at Styx. Currently in use at Spotify as a complement to the underlying job processing frameworks. https://github.com/spotify/styx

Looks like it only support cron expressions and no timezones. Moreoever, I don't plan to deploy a Kubernetes cluster in the short term (looks like to be a huge work to have something ready for production). I'm primarily looking for something I can install on the existing systems.

Re: Ask HN: What job scheduler do you use?

#7

Post by pmchorus » Sat, Apr 08, 2017, 11:36 AM UTC

paktek123 wrote: post #2

https://mesos.github.io/chronos/

I have no Mesos cluster running for now. I'm looking for something I can use on the existing infrastructure. From what I read, it does not support timezones.

Re: Ask HN: What job scheduler do you use?

#8

Post by [deleted] » Sat, Apr 08, 2017, 12:59 PM UTC

[deleted]

Re: Ask HN: What job scheduler do you use?

#9

Post by anilgulecha » Sat, Apr 08, 2017, 1:09 PM UTC

Consider AWS lambda. It's not a monolithic system, but may work depending on your use-case.

Re: Ask HN: What job scheduler do you use?

#10

Post by usernam3 » Sat, Apr 08, 2017, 2:11 PM UTC

For php laravel projects are using this service:

- https://github.com/cronboy/cronboy

- https://cronboy.com/

It doesn't fit needs of @pmchorus company, but is useful for simpler cases when you need something more efficient than cron and more accurate than queue delayed jobs. (Example: we use it for sending triggered email deliveries, where batch processing of letters using cron is not possible, because of amount of letters and cron schedule limit)

The idea is that you schedule job on specified time and cronboy invokes it via http and execute it on your server.

Features:

- Seconds accuracy of execution

- Statistics on invoked jobs

- Secure execution of your code

- Retrying if job failed

- Laravel friendly facade api