Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Plan – Cron jobs in Python

github.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Plan – Cron jobs in Python

#11

Post by Walkman »

asimjalis wrote: post #8

Here is some brainstorming on use cases. 1. You can validate cron jobs and programmatically change or reject some of them. Or maybe just offer a small set of services as a cron job. 2. Once you have a security model you can expose this as a web service and offer cron jobs over the web. 3. You could extend this API to use different implementations of cron. Instead of calling the actual cron you could implement cron in…

1. crontab -e command automatically recognize if you have a syntax problem in the newly edited file:

    $ crontab -e
    crontab: installing new crontab
    "/tmp/crontab.QhkOLb/crontab":25: bad minute
    errors in crontab file, can't install.
    Do you want to retry the same edit? (y/n)
2. See better solutions (like Celery) below.

Re: Plan – Cron jobs in Python

#13

Post by est »

Walkman wrote: post #3

I don't get it. What's so hard about cron jobs that you have to write a wrapper for it? It is easy to understand if you just read the crontab file comment at the top. There is just two commands: crontab -l for listing crontab -e for editing. What else do you need? With this, instead of learning crontab simple syntax, you need to: - install this package - learn its API - run it and still edit the crontab file :D seems…

How about dynamic periodic tasks?

For instance, server side exponential backoff.

Re: Plan – Cron jobs in Python

#14

Post by talideon »

I'd typically deal with this by exporting the functions to be ran as entry points in the the setup.py file, and then use whatever configuration management tool I'm using (Ansible, Puppet, whatever) to manage the crontab file in /etc/cron.d. Ansible's cron module makes this especially easy.

Re: Plan – Cron jobs in Python

#15

Post by noselasd »

Walkman wrote: post #3

I don't get it. What's so hard about cron jobs that you have to write a wrapper for it? It is easy to understand if you just read the crontab file comment at the top. There is just two commands: crontab -l for listing crontab -e for editing. What else do you need? With this, instead of learning crontab simple syntax, you need to: - install this package - learn its API - run it and still edit the crontab file :D seems…

This approach has a human in the loop. There's plenty of use cases where you want a program to edit cron jobs.

Re: Plan – Cron jobs in Python

#16

Post by IgorPartola »

I see the value in this: it provides a nice programmable interface to cron. Unlike what you might think at first, cron has some really interesting idiosyncrasies that you might not appreciate at first. For example, if your filename in /etc/cron.d/ has a "." character in it, it will be silently skipped [1]. Also, files in /etc/cron.d/ will not inherit environmental variables from /etc/crontab. Consider how many hours have been wasted "debugging" these issues.

That said, cron is easy to abuse as well. The example of providing exponential backoff is kind of silly: there are better tools for the job, including Celery. Cron also does not have seconds precision which is a bummer for lots of short-lived jobs. Lastly, cron only runs jobs at the specified time. If you adjust the system clock, or shut the system down, your tasks might be skipped. anacron [2] might be a better solution in some ways. Lastly, all implementations of cron that I am aware of don't have a great ability to make sure tasks actually run and finish, don't have retries built-in, and have a horrible interface for reporting errors (any output of the script is emailed to root@localhost from root@localhost).

Based on the above, you might think I am saying to never use cron. That's not at all true. Cron is extremely useful as a systems tool. If you package your software in the distro's package format (you should at least give this a try to see how the system is put together), cron is pretty much your only safe choice. Also, as long as you follow the rules cron sets up, and know of its potential issues, cron will be the workhorse that gets things done. Tasks like log rotation, periodic sanity checks, etc. are great for this. If you are writing a web application and are already using Python though, look at more sophisticated options such as Celery before using cron directly.

[1] http://manpages.ubuntu.com/manpages/maverick/en/man8/cron.8.... and https://bugs.launchpad.net/ubuntu/+source/cron/+bug/706565

[2] http://en.wikipedia.org/wiki/Anacron

Re: Plan – Cron jobs in Python

#17

Post by crdoconnor »

chhantyal wrote: post #2

And if you are already using Celery, it has crontab feature http://celery.readthedocs.org/en/latest/userguide/periodic-t...

Celery beat is pretty much exactly the same thing as cron except it's another service to configure and worry about keeping up and running. And it uses 35 MB of RAM instead of...well, I guess cron uses a few kilobytes.

I can't see the point of it at all.

When I want to kick off celery tasks I just have a cron job execute a django shell command.

Re: Plan – Cron jobs in Python

#18

Post by falcolas »

gtuckerkellogg wrote: post #10
jocmeh wrote: post #5

Earlier quoted context omitted.

Yeah, we are using that as well. It's nice because it hooks up to your Django admin and people with the right permissions can reschedule stuff. I don't see any use case for Plan, it's just complicating an easy task.

I can see one use case for this family of tools: if the same script could generate either crontab files or launchd agents, so it could generate scheduling files for MacOSX, where crontab is discouraged.

MacOSX uses launchd as its crontab, but it retains the crontab interface.

Re: Plan – Cron jobs in Python

#19

Post by icebraining »

crdoconnor wrote: post #17
chhantyal wrote: post #2

And if you are already using Celery, it has crontab feature http://celery.readthedocs.org/en/latest/userguide/periodic-t...

Celery beat is pretty much exactly the same thing as cron except it's another service to configure and worry about keeping up and running. And it uses 35 MB of RAM instead of...well, I guess cron uses a few kilobytes. I can't see the point of it at all. When I want to kick off celery tasks I just have a cron job execute a django shell command.

I guess cron uses a few kilobytes.

A full megabyte on my machine, actually! Still a long way from 35MBs, though.

Post reply on HN