Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cron in production is a double-edged sword

orchestrate.io

Post reply on HN
Print view View original

31–40 of 58 posts

Re: Cron in production is a double-edged sword

#31

Post by INTPenis »

The problem isn't cron, cron is just a dumb execution tool.

The problem is that we don't have any way of alerting our monitoring systems from a cron job.

This is exactly what I've been implementing, a simple curl API call to our monitoring system when a cron job has run is all that we need. This puts the monitoring of cron into the same sphere as all other monitoring and puts the alert on a webpage where it can be found eventually by our 2nd line or our on-call personnel, instead of in someones mailbox.

Edit: And you don't need a fancy REST based API for your monitoring system to do this, ye ol' nagios agent could do it with some hacks.

The hard part is having the discipline to fix all your cron jobs in this way, but adding || true is already tantamount to this.

Re: Cron in production is a double-edged sword

#32

Post by bbrazil »

INTPenis wrote: post #31

The problem isn't cron, cron is just a dumb execution tool. The problem is that we don't have any way of alerting our monitoring systems from a cron job. This is exactly what I've been implementing, a simple curl API call to our monitoring system when a cron job has run is all that we need. This puts the monitoring of cron into the same sphere as all other monitoring and puts the alert on a webpage where it can be fo…

This is basically the approach we take with Prometheus, with the option to add in additional stats like duration and processed records too.

http://www.robustperception.io/monitoring-batch-jobs-in-pyth... is the full Python version, and the simple version is a bash one-liner too.

Re: Cron in production is a double-edged sword

#33

Post by KaiserPro »

NightMKoder wrote: post #17

Something we've found to be fairly lightweight (compared to e.g. Chronos), but incredibly featureful is using Jenkins (the CI server) as a cron runner. We use http://docs.openstack.org/infra/jenkins-job-builder/ to configure it at deploy-time so it lives as part of the deploy rather than system config. Here's a small list of things we're getting out of it: - concurrent run protection (& queue management via https://w…

Yup, we do the same.

We made sure that jenkins doesn't fiddle with the environment, so that everything was derived from the various networked user accounts.

using @hourly, it spreads the load evenly over the hour to even out resource starvation spikes.

We have jenkin's job builder(and yaml) in a git repo to make sure that the delicate snowflake that is jenkins is repeatable.

Re: Cron in production is a double-edged sword

#34

Post by sebw »

No one mentionned Rundeck: http://rundeck.org/

I've been using it for two years now. This has replaced cron on about 200 nodes.

Not only it does cron, but also helps deploying artefacts (integrated with Jenkins) through simple forms. We now have ops with 0 experience in Linux deploying code.

Re: Cron in production is a double-edged sword

#36

Post by rnhmjoj »

vidarh wrote: post #18

Most of what he's writing about, and much more, is made substantially easier with systemd timers. E.g. want errors to cause e-mails, but everything else to just go to logs? Use a timer to activate a service, and make systemd activate another service on failure. Want to avoid double execution? That's the default (timers are usually used to activate another unit, as long as that unit doesn't start something that double…

I agree: I recently moved my scripts from crontab to systemd timers and there is no going back. Finally I have a proper way to debug and log. Also on NixOS I can have the unit file and timer generated in very few lines. Look at this one for example:

    "xkcd" = {
       description = "send latest xkcd comic"; 
       wants = [ "network.target" ]; 
       startAt = "Mon,Wed,Fri *:0/30"; 
  
       path = with pkgs; [ telegram-cli ];   
       serviceConfig = { 
         User = "rnhmjoj"; 
         Type = "oneshot"; 
         ExecStart = "${cabal}/bin/xkcd"; 
       };
     } // basicEnv;

Re: Cron in production is a double-edged sword

#37

Post by zimbatm »

Having local mailboxes in each server is not really useful in a cloud setup with hundreds of machines. But it's not a reason to silence the output; something bad might happen and only stdout/stderr might give you an anwer of what exactly is going wrong.

Instead use https://github.com/zimbatm/logmail. It's a `sendmail` replacement that forwards everything to syslog. Then forward all your syslogs to a central place an you can capture and analyze these messages.

Re: Cron in production is a double-edged sword

#40

Post by radiowave »

sebw wrote: post #34

No one mentionned Rundeck: http://rundeck.org/ I've been using it for two years now. This has replaced cron on about 200 nodes. Not only it does cron, but also helps deploying artefacts (integrated with Jenkins) through simple forms. We now have ops with 0 experience in Linux deploying code.

+1. Also, I replaced all my nagios event handlers with rundeck jobs, so nagios just calls the rundeck API. I get a full audit trail of when the job ran, with what parameters, how long it took, and its outcome.
Post reply on HN