Live data from Hacker News

Red Engine: modern scheduling framework for Python applications

red-engine.readthedocs.io

41–50 of 55 posts

Re: Red Engine: modern scheduling framework for Python applications

#41

Off topic but, Many scheduling system I’ve worked with have this weird tendency to run when deployed and then every time it’s time for it to run. I’ve had this happen with kubernetes, Scheduled Quries in GCP big query and a few other systems. That seems like absolute madness. Why would anything do that?

Airflow does this. Madness!

Re: Red Engine: modern scheduling framework for Python applications

#42

Earlier quoted context omitted.

Would you have a recommendation for an easy to use Python scheduler with such a feature for a personal project?

APScheduler

It looks pretty good. Thanks!

Re: Red Engine: modern scheduling framework for Python applications

#43
post #27

Earlier quoted context omitted.

Other schedulers have a durable database of attempted runs. This doesn't seem to have anything like that.

Would you have a recommendation for an easy to use Python scheduler with such a feature for a personal project?

I think you can create ephemeral timers with Systemd if you're on Linux.

Re: Red Engine: modern scheduling framework for Python applications

#44

Off topic but, Many scheduling system I’ve worked with have this weird tendency to run when deployed and then every time it’s time for it to run. I’ve had this happen with kubernetes, Scheduled Quries in GCP big query and a few other systems. That seems like absolute madness. Why would anything do that?

I imagine it's to ensure it will run ASAP in case the scheduler crashes, power failure or whatever. Systemd has a similar feature "Persistence" for timers but it's configurable.

Re: Red Engine: modern scheduling framework for Python applications

#45
post #27

Earlier quoted context omitted.

Other schedulers have a durable database of attempted runs. This doesn't seem to have anything like that.

Would you have a recommendation for an easy to use Python scheduler with such a feature for a personal project?

Celery Beat is pretty good

Re: Red Engine: modern scheduling framework for Python applications

#46
post #27
post #22

Earlier quoted context omitted.

If it took less than 5 minutes to boot then I can't see why it wouldn't work. How would that work for other schedulers? Also, if a server reboots that's quite bad all round anyway. Hopefully you'd be notified directly.

Other schedulers have a durable database of attempted runs. This doesn't seem to have anything like that.

From its docs, I understand that it has a task status logging layer[1] which can persist to SQL/Mongo (using another framework called Red Bird[2]). I haven't tried them out though.

[1]: https://red-engine.readthedocs.io/en/stable/tutorial/basic.h...

[2]: https://red-bird.readthedocs.io/en/latest/

Re: Red Engine: modern scheduling framework for Python applications

#47
post #29

How does it handle state and restarts? What happens if a job is scheduled to run "before 10am", then the entire server restarts at 9:55am, will it try to run that same job again when it boots back up?

It will run it if the scheduler is called before 10am according to the docs - at runtime the conditions must be met.

Right but what if it already ran? Should the jobs be written such that they are tolerant to re-runs?

Re: Red Engine: modern scheduling framework for Python applications

#48
post #3

> Clean: Scheduling is just plain English Ugh, no thanks. First of all, English itself is not clean; it's a messy amalgamation of special cases and inconsistent spelling rules. Second, it isn't actually English anyway. It might look like English, but it's actually a DSL that happens to correspond to English a lot of the time. English text is meant to be interpreted by humans, who understand context & connotation, and…

100%. also, why re-invent python's already pretty decent datetime/timedeltas?

and why not a proper python DSL?

  from redengine import minute, hour
  
  @app.run_every(hour + 20*minute)
  def do_first(): [...]
  
  @app.run_after(do_first)
  def do_second1(): [...]
  
  @app.run_after(do_first)
  def do_second2(): [...]
  
  @app.run_after(do_second1 and do_second2)
  def do_last(): [...]

Re: Red Engine: modern scheduling framework for Python applications

#49
post #29

Earlier quoted context omitted.

It will run it if the scheduler is called before 10am according to the docs - at runtime the conditions must be met.

Right but what if it already ran? Should the jobs be written such that they are tolerant to re-runs?

I think that regardless of the scheduling tool, it is very worthwhile putting in the effort to make your tasks idempotent. I've had tons of cases when I had to repeatedly rerun failed tasks "in anger", and was always grateful to know I made it safe to do so.

Re: Red Engine: modern scheduling framework for Python applications

#50

This fits my use case so perfectly! I have a very small internal app taking care of organizing seminar talks, calendars, email announcements, recordings of the talks, and signups. It is a single python file of less than 1500 lines and an sqlite database. This library is so perfect for taking care of scheduled events. Everything else I have found is ridiculously over-complicated of a solution.

Can you share repo to your application?

I assume that “internal” means it’s just that and not accessible.
Post reply on HN