Cron in production is a double-edged sword • Hacker News Board

Re: Cron in production is a double-edged sword

#41

Post by ck2 » Sat, Oct 17, 2015, 11:47 AM UTC

gee this is a difficult concept

how to get cron to only send important emails and not every time it runs

you think maybe you should have just used

    > /dev/null

and not

    > /dev/null 2>&1

why is this a full blog post?

Re: Cron in production is a double-edged sword

#42

Post by eru » Sat, Oct 17, 2015, 12:14 PM UTC

rnhmjoj wrote: post #36

vidarh wrote: post #18

Most of what he's writing about, and much more, is made substantially easier with systemd timers. E.g. want errors to cause e-mails, but everything else to just go to logs? Use a timer to activate a service, and make systemd activate another service on failure. Want to avoid double execution? That's the default (timers are usually used to activate another unit, as long as that unit doesn't start something that double…

I agree: I recently moved my scripts from crontab to systemd timers and there is no going back. Finally I have a proper way to debug and log. Also on NixOS I can have the unit file and timer generated in very few lines. Look at this one for example: "xkcd" = { description = "send latest xkcd comic"; wants = [ "network.target" ]; startAt = "Mon,Wed,Fri *:0/30"; path = with pkgs; [ telegram-cli ]; serviceConfig = { Use…

It's a fun example, but for webcomics RSS is probably the better solution than scheduling something on your machine.

Re: Cron in production is a double-edged sword

#43

Post by St-Clock » Sat, Oct 17, 2015, 12:19 PM UTC

nathancahill wrote: post #9

DannoHung wrote: post #8

Is there any good open source distributed scheduler that blends both timer based tasks and event based tasks? Chronos is the only one I'm aware of, but I don't believe it supports event based tasks.

Celery is a task queue that supports scheduling as well, if you're using Python. Backed by several different brokers including Redis.

Shameless plug: we replaced most of our cron jobs with celery and django celery fulldbresult[1], which provides more info and features than the default django-celery integration:

- Save result as json, which is queryable in the database

- Save enough info in the task result to retry the task from the result.

- Retry a task from its result in Django Admin

- Run a periodic task now (e.g., to test your cron task)

[1] https://github.com/resulto-admin/django-celery-fulldbresult

Re: Cron in production is a double-edged sword

#44

Post by vacri » Sat, Oct 17, 2015, 1:00 PM UTC

Another cron trick is to use chronic/cronic before your command. It silences the command except for error states - cron likes to report any text it sees, which you don't want for non-error states. It also detects errors better than just assuming all errors happen on STDERR.

Re: Cron in production is a double-edged sword

#45

Post by geggam » Sat, Oct 17, 2015, 2:03 PM UTC

Nice blog explaining how someone cant use cron. Doesn't mean the rest of us are that incapable.

Re: Cron in production is a double-edged sword

#46

Post by smackfu » Sat, Oct 17, 2015, 2:14 PM UTC

geggam wrote: post #45

Nice blog explaining how someone cant use cron. Doesn't mean the rest of us are that incapable.

I admire anyone who puts their learning process out in public to help other people.

Re: Cron in production is a double-edged sword

#47

Post by psior » Sat, Oct 17, 2015, 2:38 PM UTC

Very interesting discussion. I've been doing some related work where I needed to run some tasks in a non-overlapping way and while flock was an initial option I later moved to redis queue (ie rpush & blpop mix) to guarantee a certain (and needed) order of execution. This is mixed with a 'send email in case of error' check and so far is doing fine, though I'll definitely look into Jenkins if I ever feel this current approach proves not to be reliable enough.

Re: Cron in production is a double-edged sword

#48

Post by rbc » Sat, Oct 17, 2015, 2:43 PM UTC

CFEngine also provides a scheduling capability that can be used in conjunction with other factors using boolean expressions. Something like "Run at midnight on Saturday if you are a production linux server." The splaytime parameter can spread out the execution of a command across a cluster based on its name hash.

Re: Cron in production is a double-edged sword

#49

Post by rbc » Sat, Oct 17, 2015, 2:58 PM UTC

Something else. The cron service is a one hit wonder. All it does is schedule. It places responsibility for handling output and setting semaphores for use by other applications to the person who wrote the command called by cron. You can't really blame cron if the command/script doesn't do these things. You just need to look to another type of scheduler/batch facility that provides a richer feature set for handling workflow, monitoring and reporting.

Re: Cron in production is a double-edged sword

#50

Post by liquidise » Sat, Oct 17, 2015, 3:34 PM UTC

This is not an argument against cron. It is a demonstration of people not abstracting code. One of the thousands i've come across.

Take all of the features he mentions, and abstract the to a launch_from_cron.sh file. Make that file accept a script path as an argument and viola! All of the safety added to cron without the need for code duplication or these massive overhead solutions listed in these comments.