Cron in production is a double-edged sword • Hacker News Board

Re: Cron in production is a double-edged sword

#21

Post by darylteo » Sat, Oct 17, 2015, 2:28 AM UTC

NightMKoder wrote: post #17

Something we've found to be fairly lightweight (compared to e.g. Chronos), but incredibly featureful is using Jenkins (the CI server) as a cron runner. We use http://docs.openstack.org/infra/jenkins-job-builder/ to configure it at deploy-time so it lives as part of the deploy rather than system config. Here's a small list of things we're getting out of it: - concurrent run protection (& queue management via https://w…

For those preferring a self-hosted oss monitoring solution, Jenkins is a good multi-purpose choice (it does more than continuous integration!).

I inherited a legacy application with tons of cron jobs running scripts on the production server. Instead of risking moving our jobs to jenkins, we're simply using jenkin's post endpoint to post job results from the cron jobs themselves. It's not perfect, and doesn't give us all the goodies listed above, but it does give us more visibility on the jobs themselves until we can move them all off reliably. +1 from me if you are in a similar situation.

Re: Cron in production is a double-edged sword

#22

Post by josnyder » Sat, Oct 17, 2015, 2:31 AM UTC

I work for Yelp, and we use cron for purposes similar to those mentioned in this article, mostly synchronizing small bits of configuration or data that we want local to the machine. We're heavy Puppet users, and we made a module to assist us in the management of our crons [1]. If you're a Puppet shop, I highly recommend checking it out. It provides answers to each of the problems mentioned in the article, often using the same mechanisms. I especially like its integration with Sensu, which we use for monitoring the jobs.

We've found that deploying cronjobs onto individual hosts is quite powerful, and helps us fill a niche between configuration management tools (like Puppet) and specialized coprocesses (like Smartstack). We have cronjobs for downloading code deploys, showing Sensu state within the motd, reconfiguring daemons (especially the Smartstack ones), and (of course) cleaning up unused data.

Of course, there's also the separate problem of scheduling and coordinating tasks across an entire cluster. In most cases we don't use our cron daemons for this, although we do have some jobs that run on multiple hosts and enforce mutual exclusion by grabbing a lock in Zookeeper.

[1] https://github.com/Yelp/puppet-cron#puppet-cron

Re: Cron in production is a double-edged sword

#23

Post by eyepulp » Sat, Oct 17, 2015, 2:51 AM UTC

cuu508 wrote: post #11

nathancahill wrote: post #3

I've been using Dead Man's Snitch[0] in production for a few years. It's been a life saver. Not affiliated, just a happy customer. [0] https://deadmanssnitch.com/

Shameless plug: https://healthchecks.io Same idea, open source

Healthchecks.io looks really interesting, both because it's an open source django project and because I was disappointed with Dead Man's Snitch. DMS forces me to live within their timing for running checks -- If you have something that has to occur @ 3am every morning, you won't know it failed until midnight UTC later that day, or when a customer calls to complain.

Healthchecks handles this a lot more sensibly. I might throw it on a linode and give it a shot. Thanks for releasing it.

Re: Cron in production is a double-edged sword

#24

Post by ausjke » Sat, Oct 17, 2015, 3:01 AM UTC

Great read and definitely will keep this in my toolbox, the whole article is explaining why the below good when you need use cron:

15 * * * * ( flock -w 0 200 && sleep `perl -e 'print int(rand(60))'` && nice /command/to/run && date > /var/run/last_successful_run ) 2>&1 200> /var/run/cron_job_lock | while read line ; echo `date` "$line" ; done > /path/to/the/log || true

Re: Cron in production is a double-edged sword

#25

Post by chishaku » Sat, Oct 17, 2015, 3:21 AM UTC

bpchaps wrote: post #7

zpidey wrote: post #2

These all seem like issues you'd run into with any task scheduler. Error emails, overloading a central resource with many tasks. Most of these aren't particular/limited to cron at all.

I dunno. Cron is particularly bad. Want a sane looking cron? You'll probably end up writing a wrapper script to handle stdout/err. Every time I deal with an annoying dev or proprietary binary, my crons turn to a total mess. Also: /home/on_a_phone/parse_today.sh `date +%Y%m%d` Will fail catastrophically because cron treats '%' as a newline character for some silly reason. Have fun troubleshooting that one! Side note -…

DATE=date +%Y%m%d

/home/on_a_phone/parse_today.sh $DATE

Re: Cron in production is a double-edged sword

#26

Post by aidenn0 » Sat, Oct 17, 2015, 4:01 AM UTC

Karunamon wrote: post #14

aidenn0 wrote: post #10

I use Jenkins instead of cron. I get an rss feed of processes that exited with non-zero, it captures the output but doesn't e-mail it to me. This is totally not what it's designed for, but it is closer to what I want than cron is.

That's actually a pretty damn brilliant use of a CI system. Can it be distributed, though? All timed jobs running from one box screams "single point of failure"

It's as distributed or not as cron.

Re: Cron in production is a double-edged sword

#27

Post by pinterface » Sat, Oct 17, 2015, 4:56 AM UTC

I've been pretty happy with shush[1], which is a similar script that helps with a lot of this--including randomdelays, locking to avoid overlapping runs, e-mailing only on errors (or other criteria as you see fit), and so forth.

[1] http://web.taranis.org/shush/

Re: Cron in production is a double-edged sword

#28

Post by Karunamon » Sat, Oct 17, 2015, 5:59 AM UTC

aidenn0 wrote: post #26

Karunamon wrote: post #14

Earlier quoted context omitted.

That's actually a pretty damn brilliant use of a CI system. Can it be distributed, though? All timed jobs running from one box screams "single point of failure"

It's as distributed or not as cron.

I meant, if 5 boxes run cron and one box blows up, only the jobs on that box are affected. If one box is running all jobs and it blows up, all jobs are affected.

Re: Cron in production is a double-edged sword

#29

Post by gluelogic » Sat, Oct 17, 2015, 6:01 AM UTC

nathancahill wrote: post #3

I've been using Dead Man's Snitch[0] in production for a few years. It's been a life saver. Not affiliated, just a happy customer. [0] https://deadmanssnitch.com/

I'll throw in a vote for DMS. I use it at work to verify that our cron jobs ran successfully. Dead simple and very effective.

Re: Cron in production is a double-edged sword

#30

Post by paulfurtado » Sat, Oct 17, 2015, 6:32 AM UTC

aidenn0 wrote: post #26

Karunamon wrote: post #14

Earlier quoted context omitted.

That's actually a pretty damn brilliant use of a CI system. Can it be distributed, though? All timed jobs running from one box screams "single point of failure"

It's as distributed or not as cron.

That's not really true. With Jenkins, the Jenkins master controls 100% of the execution of jobs on the slaves and Jenkins cannot be multi-master. With cron, each host is responsible for executing its own cron jobs, removing the single point of failure.