Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cron in production is a double-edged sword

orchestrate.io

Post reply on HN
Print view View original

51–58 of 58 posts

Re: Cron in production is a double-edged sword

#51

Post by willcodeforfoo »

NightMKoder wrote: post #17

Something we've found to be fairly lightweight (compared to e.g. Chronos), but incredibly featureful is using Jenkins (the CI server) as a cron runner. We use http://docs.openstack.org/infra/jenkins-job-builder/ to configure it at deploy-time so it lives as part of the deploy rather than system config. Here's a small list of things we're getting out of it: - concurrent run protection (& queue management via https://w…

We use Jenkins for a cron-replacement too. We've noticed all the benefits you mention plus it's dead easy for others in the organization to (re)run tasks, even with different parameters.

Re: Cron in production is a double-edged sword

#52

Post by andmarios »

NightMKoder wrote: post #17

Something we've found to be fairly lightweight (compared to e.g. Chronos), but incredibly featureful is using Jenkins (the CI server) as a cron runner. We use http://docs.openstack.org/infra/jenkins-job-builder/ to configure it at deploy-time so it lives as part of the deploy rather than system config. Here's a small list of things we're getting out of it: - concurrent run protection (& queue management via https://w…

Jenkins is nice but you should be careful.

Once I had a job that went stray and got the disk full with logs. Since Jenkins couldn't write to the disk anymore, it stopped working completely and thus no jobs and more importantly no notifications. Funny thing, there was one job to monitor the free disk space but the stray app wrote ~100GB in less than 15 minutes (damn SSDs :p).

Another time (times actually), I had the OOM killer kill a jenkins related process. Being a JVM based app and starting with about 1GB of RAM use, doesn't help I guess. This lead Jenkins to hang on a job; timeout didn't work, I couldn't even stop the job manually. Other jobs wouldn't start and no notifications would be sent again.

Re: Cron in production is a double-edged sword

#53

Post by nathancahill »

eyepulp wrote: post #23
cuu508 wrote: post #11

Earlier quoted context omitted.

Shameless plug: https://healthchecks.io Same idea, open source

Healthchecks.io looks really interesting, both because it's an open source django project and because I was disappointed with Dead Man's Snitch. DMS forces me to live within their timing for running checks -- If you have something that has to occur @ 3am every morning, you won't know it failed until midnight UTC later that day, or when a customer calls to complain. Healthchecks handles this a lot more sensibly. I mig…

Wow, that's awesome. That really is the biggest problem with DMS. I asked them about that feature a couple years ago, they said it was on the roadmap. Might ping them again.

Re: Cron in production is a double-edged sword

#54

Post by weaksauce »

liquidise wrote: post #50

This is not an argument against cron. It is a demonstration of people not abstracting code. One of the thousands i've come across. Take all of the features he mentions, and abstract the to a launch_from_cron.sh file. Make that file accept a script path as an argument and viola! All of the safety added to cron without the need for code duplication or these massive overhead solutions listed in these comments.

did you not see the cron script at the end of the article? the author does exactly this.

Re: Cron in production is a double-edged sword

#55

Post by 2bluesc »

vidarh wrote: post #18

Most of what he's writing about, and much more, is made substantially easier with systemd timers. E.g. want errors to cause e-mails, but everything else to just go to logs? Use a timer to activate a service, and make systemd activate another service on failure. Want to avoid double execution? That's the default (timers are usually used to activate another unit, as long as that unit doesn't start something that double…

This. While certain crowds like to hate on systemd, the many features beyond init are lost in the noise for the casual observers. I love systemd timers.

The biggest shortcoming with systemd timers, is that it doesn't have an easy way to notify admins of failures like standard cron does.

I tried to hack around this[0], but it still feels wrong.

[0] https://github.com/kylemanna/systemd-utils#scripts

Re: Cron in production is a double-edged sword

#56

Post by bpchaps »

fsaintjacques wrote: post #20
bpchaps wrote: post #7

Earlier quoted context omitted.

I dunno. Cron is particularly bad. Want a sane looking cron? You'll probably end up writing a wrapper script to handle stdout/err. Every time I deal with an annoying dev or proprietary binary, my crons turn to a total mess. Also: /home/on_a_phone/parse_today.sh `date +%Y%m%d` Will fail catastrophically because cron treats '%' as a newline character for some silly reason. Have fun troubleshooting that one! Side note -…

You can solve said problem by having a semi decent CLI api with defaults, e.g. "--date= defaults to $TODAY".

That's seriously, seriously easier said than done in about 10% of cases that this sort of thing comes up. Especially when dealing with awful vendor code.

What about an application that takes an arbitrary date as input? Keep in mind that we're talking about production-leven infrastructures with many potentially many thousands of servers that might have 10 different distros with many thousands of differences between each machine, so falling back to "Just install X" isn't a possibility.

Then again, there's something to be said for vetting an application/script for prod-use on its "cronability". I don't think that's the point you were going for, though.

Re: Cron in production is a double-edged sword

#57

Post by aidenn0 »

Karunamon wrote: post #28
aidenn0 wrote: post #26

Earlier quoted context omitted.

It's as distributed or not as cron.

I meant, if 5 boxes run cron and one box blows up, only the jobs on that box are affected. If one box is running all jobs and it blows up, all jobs are affected.

If 5 boxes run jenkins and one box blows up, only the jobs on that box are affected.

Re: Cron in production is a double-edged sword

#58

Post by hbogert »

Excessive use of crons is a devops (hate the word) smell. You get reliant on their side-effects and to migrate to other solutions you need enormous amounts of testing and legacy interfaces. The most obvious downside to a cron is the at least 1m interval. On average you are waiting 30s for something which already should be there. Of course it's perfect for things like reporting which make sense for certain intervals. Using it for mail queues and stuff.. bad times.
Post reply on HN