Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cron in production is a double-edged sword

orchestrate.io

Post reply on HN
Print view View original

11–20 of 58 posts

Re: Cron in production is a double-edged sword

#13

Post by lstamour »

patio11 wrote: post #6
nathancahill wrote: post #3

I've been using Dead Man's Snitch[0] in production for a few years. It's been a life saver. Not affiliated, just a happy customer. [0] https://deadmanssnitch.com/

Seconded. DMS is the easiest thing to just drop-in on the Nth cron job you add. Eventually you might need something more complicated for monitoring/outages/etc, and that something is probably either a whole lot of Nagios and bailing wire and/or PagerDuty, but DMS is perfect for "I really need Tarsnap backups to not just silently fail." I also end up creating a lot of Twilio scripts which are either positive control o…

This reminds me of https://docs.google.com/a/gravitant.com/document/d/199PqyG3U... on how you should only wake up engineers when there really is a problem. I'd suggest logging based on error messages -- though I get it, if a problem occurs upstream, you wouldn't know it unless you'd polled for it too, as a data point. HN comments on that doc at: https://news.ycombinator.com/item?id=8450147

Re: Cron in production is a double-edged sword

#14

Post by Karunamon »

aidenn0 wrote: post #10

I use Jenkins instead of cron. I get an rss feed of processes that exited with non-zero, it captures the output but doesn't e-mail it to me. This is totally not what it's designed for, but it is closer to what I want than cron is.

That's actually a pretty damn brilliant use of a CI system. Can it be distributed, though? All timed jobs running from one box screams "single point of failure"

Re: Cron in production is a double-edged sword

#15

Post by rodgerd »

DannoHung wrote: post #8

Is there any good open source distributed scheduler that blends both timer based tasks and event based tasks? Chronos is the only one I'm aware of, but I don't believe it supports event based tasks.

JobScheduler by SOS is free, distributed, supports most *ixes, Windows, and covers most of the things I want in a scheduler.

Re: Cron in production is a double-edged sword

#16

Post by stevepike »

aidenn0 wrote: post #10

I use Jenkins instead of cron. I get an rss feed of processes that exited with non-zero, it captures the output but doesn't e-mail it to me. This is totally not what it's designed for, but it is closer to what I want than cron is.

I do this too. We basically use Jenkins as "cron that the non-engineering team can read with a web gui, auto-archiving of files, configurable email notifications". It's ugly but it gets the job done.

Re: Cron in production is a double-edged sword

#17

Post by NightMKoder »

Something we've found to be fairly lightweight (compared to e.g. Chronos), but incredibly featureful is using Jenkins (the CI server) as a cron runner. We use http://docs.openstack.org/infra/jenkins-job-builder/ to configure it at deploy-time so it lives as part of the deploy rather than system config.

Here's a small list of things we're getting out of it:

- concurrent run protection (& queue management via https://wiki.jenkins-ci.org/display/JENKINS/Concurrent+Run+B... )

- load balancing (e.g. max concurrent tasks) and remote execution with jenkins slaves [sounds complicated, but really jenkins just knows how to SSH]

- job timeouts. No more hanging jobs.

- failure notifications via slack/hipchat/email/whatever. [email only on status change via https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin ]

- log/history management: rotation & compression.

- fancy scheduling: e.g. run this job once every 24h, but if it fails keep retrying in 5 minute increments (https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin ). You could also use project dependencies for pipelines, but we've been staying away from that.

- monitoring: we use the datadog reporter & alert on time since last success. Given how mature Jenkins is, this likely translates to whatever system you're using just as well.

It's worked incredibly well for us. We migrated to Jenkins from crontabs with cronwrap (https://github.com/zomo/cronwrap). We're never going back.

Re: Cron in production is a double-edged sword

#18

Post by vidarh »

Most of what he's writing about, and much more, is made substantially easier with systemd timers.

E.g. want errors to cause e-mails, but everything else to just go to logs? Use a timer to activate a service, and make systemd activate another service on failure.

Want to avoid double execution? That's the default (timers are usually used to activate another unit, as long as that unit doesn't start something that doubleforks, it won't get activated twice).

(Some) protection against thundering herd is built in: You specify the level of accuracy (default 1m), and each machine on boot will randomly select a number of seconds to offset all timers on that host with. You can set this per timer or for the entire host.

And if you're using fleet, you can use fleet to automatically re-schedule cluster-wide jobs if a machine fails.

And the journal will capture all the output and timestamp it.

systemctl list-timers will show you which timers are scheduled, when they're scheduled to run next, how long is left until then, when they ran last, how long that is ago:

     $ systemctl list-timers
    NEXT                         LEFT     LAST                         PASSED       UNIT                      
    Sat 2015-10-17 01:30:15 UTC  51s left Sat 2015-10-17 01:29:15 UTC  8s ago       motdgen.timer             
    Sat 2015-10-17 12:00:34 UTC  10h left Sat 2015-10-17 00:00:33 UTC  1h 28min ago rkt-gc.timer              
    Sun 2015-10-18 00:00:00 UTC  22h left Sat 2015-10-17 00:00:00 UTC  1h 29min ago logrotate.timer           
    Sun 2015-10-18 00:15:26 UTC  22h left Sat 2015-10-17 00:15:26 UTC  1h 13min ago systemd-tmpfiles-clean.timer
And the timer specification itself is extremely flexible. E.g. you can schedule a timer to run x seconds after a specific unit was activated, or x seconds after boot, or x seconds after the timer itself fired, or x seconds after another unit was deactivated. Or combinations.

Re: Cron in production is a double-edged sword

#20

Post by fsaintjacques »

bpchaps wrote: post #7
zpidey wrote: post #2

These all seem like issues you'd run into with any task scheduler. Error emails, overloading a central resource with many tasks. Most of these aren't particular/limited to cron at all.

I dunno. Cron is particularly bad. Want a sane looking cron? You'll probably end up writing a wrapper script to handle stdout/err. Every time I deal with an annoying dev or proprietary binary, my crons turn to a total mess. Also: /home/on_a_phone/parse_today.sh `date +%Y%m%d` Will fail catastrophically because cron treats '%' as a newline character for some silly reason. Have fun troubleshooting that one! Side note -…

You can solve said problem by having a semi decent CLI api with defaults, e.g. "--date= defaults to $TODAY".
Post reply on HN