I'm using APScheduler for most of my scheduling tasks.
Does Red Engine integrate with asyncio? When searching the docs for this keyword no hits showed up.
Red Engine: modern scheduling framework for Python applications
21–30 of 55 posts
Re: Red Engine: modern scheduling framework for Python applications
#22How does it handle state and restarts? What happens if a job is scheduled to run "before 10am", then the entire server restarts at 9:55am, will it try to run that same job again when it boots back up?
If it took less than 5 minutes to boot then I can't see why it wouldn't work.
How would that work for other schedulers? Also, if a server reboots that's quite bad all round anyway. Hopefully you'd be notified directly.
Re: Red Engine: modern scheduling framework for Python applications
#23> Clean: Scheduling is just plain English Ugh, no thanks. First of all, English itself is not clean; it's a messy amalgamation of special cases and inconsistent spelling rules. Second, it isn't actually English anyway. It might look like English, but it's actually a DSL that happens to correspond to English a lot of the time. English text is meant to be interpreted by humans, who understand context & connotation, and…
It’s plain English like SQL is plain English. It uses English words that’s kind-of make sense to a novice, but that doesn’t stop it being a new programming language.
Re: Red Engine: modern scheduling framework for Python applications
#24If you're going to build a framework like this, please please put your consistency behavior and retry behavior at the top of your docs.
Yes, 100% this. Too many frameworks claim that they are "easy" just to find out they left out things that other older frameworks have already solved.
While we are on this: Do you know of any task scheduler framework that is similar to celery, but has better guarantees around task execution than what acks_late= True gives you?
I always find myself building a system that stores the really important Tasks in Postgres so that I can recover from anything in the broker or celery crashing. What i use celery for is just scheduling these Tasks by creating a celery job with the Postgres Job ID as the parameter.
Then, to detect if something went horribly wrong, I have a sweeping job that checks if any job in Postgres has not run in celery for some reason. If that is the case, we just re-queue the job.
Re: Red Engine: modern scheduling framework for Python applications
#25If you're going to build a framework like this, please please put your consistency behavior and retry behavior at the top of your docs.
Yes, 100% this. Too many frameworks claim that they are "easy" just to find out they left out things that other older frameworks have already solved. While we are on this: Do you know of any task scheduler framework that is similar to celery, but has better guarantees around task execution than what acks_late= True gives you? I always find myself building a system that stores the really important Tasks in Postgres so…
After reading the linked article in more detail, I might have misinterpreted things a bit, but my concerns are still related :)
Re: Red Engine: modern scheduling framework for Python applications
#26> Clean: Scheduling is just plain English Ugh, no thanks. First of all, English itself is not clean; it's a messy amalgamation of special cases and inconsistent spelling rules. Second, it isn't actually English anyway. It might look like English, but it's actually a DSL that happens to correspond to English a lot of the time. English text is meant to be interpreted by humans, who understand context & connotation, and…
> See also: SQL, Python
I do not see what is wrong with SQL and I definitely do not see what is wrong with Python.
Re: Red Engine: modern scheduling framework for Python applications
#27How does it handle state and restarts? What happens if a job is scheduled to run "before 10am", then the entire server restarts at 9:55am, will it try to run that same job again when it boots back up?
If it took less than 5 minutes to boot then I can't see why it wouldn't work. How would that work for other schedulers? Also, if a server reboots that's quite bad all round anyway. Hopefully you'd be notified directly.
Other schedulers have a durable database of attempted runs. This doesn't seem to have anything like that.
Re: Red Engine: modern scheduling framework for Python applications
#28Is this meant for use instead of something like Luigi?
Re: Red Engine: modern scheduling framework for Python applications
#29How does it handle state and restarts? What happens if a job is scheduled to run "before 10am", then the entire server restarts at 9:55am, will it try to run that same job again when it boots back up?
It will run it if the scheduler is called before 10am according to the docs - at runtime the conditions must be met.
Re: Red Engine: modern scheduling framework for Python applications
#30Using Python decorators is a very strange choice for constructing computational graphs. Then again, I don't really use Python for this kind of thing, I roll my own solutions (of just functions being composed together).
One big issue I have with the proposed approach is that it's very difficult for me to see at a glance the actual compute graph. I suppose you can build some tools to visualize it from the DSL in the decorator call, but I'd much rather be able to see this directly in code, with no weird magic, so that I can very easily interpret and update it if need be.