Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing Cron Scripts Reliably at Scale

slack.engineering

Post reply on HN
Print view View original

51–60 of 100 posts

Re: Executing Cron Scripts Reliably at Scale

#51

Post by Supermancho »

Scubabear68 wrote: post #43
dimgl wrote: post #23

Earlier quoted context omitted.

Why is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.

Cron scaling? Seriously? By definition cron does not scale, it is account by account per VM/machine with no rhyme or reason.

Alternately, if you define cron as a starting/startup point, an expectation when scaling, then simple format for a job queue, the context of the post is understood.

Re: Executing Cron Scripts Reliably at Scale

#52

Post by paxys »

whorleater wrote: post #20

I would be curious why kube cron jobs didn't seem to fit the bill, my favorite part of these posts are when they have a section hinting that they explored other options picked specific tradeoffs

Spinning up a new Kubernetes pod for every single job run is a very expensive and wasteful operation, starting at least in the order of seconds (usually more) vs just milliseconds for a new process in an already hot environment.

Re: Executing Cron Scripts Reliably at Scale

#53

Post by zerbinxx »

reactordev wrote: post #39
dimgl wrote: post #23

Earlier quoted context omitted.

Why is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.

It’s no longer simple when you have platform code to prevent nodes from disappearing or dying on the minute on a kubernetes cluster the size of slack. A triggered pulse event stream would have done the trick to invoke a lambda or call code for every “thing” that needed a beat. Kubernetes comes with a scheduler…

This is my take as well, just because you can get away with something doesn’t necessarily make it desirable to maintain or extend, and I honestly cannot imagine the effort in terms of labor hours that you’d have to go through to develop something like this compared to just plugging in an off-the-shelf scheduler to something slightly more sophisticated like k8s or even just a worker-and-queue system. When you’re talking about platform engineering to solve a problem that a relatively extensible Celery service could do (and have tests and such), I have no idea how the former could be “less work” or cheaper in the long haul.

Re: Executing Cron Scripts Reliably at Scale

#54

Post by magarnicle »

sciurus wrote: post #44
ohnoesjmr wrote: post #28

Earlier quoted context omitted.

Can you name some? I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)

I almost hate to say it, but Jenkins.

That's what I did. I know it was the right solution because I happily keep adding more and more scripts to run. I didn't do that when I was using cron jobs.

Plus, Jenkins has a few nice extensions to the crontab, including setting a timezone and using "H" to spread job execution load.

Re: Executing Cron Scripts Reliably at Scale

#55

Post by zerbinxx »

paxys wrote: post #52
whorleater wrote: post #20

I would be curious why kube cron jobs didn't seem to fit the bill, my favorite part of these posts are when they have a section hinting that they explored other options picked specific tradeoffs

Spinning up a new Kubernetes pod for every single job run is a very expensive and wasteful operation, starting at least in the order of seconds (usually more) vs just milliseconds for a new process in an already hot environment.

Sure, but if you need that thing to run every hour for a few seconds, then seconds aren’t really the limiting factor. I don’t doubt that the resource management side of k8s would make it dicey at a certain volume of these things running, though, especially if they eat a lot of compute.

Re: Executing Cron Scripts Reliably at Scale

#56

Post by fukawi2 »

ohnoesjmr wrote: post #28
Scubabear68 wrote: post #21

I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.

Can you name some? I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)

Rundeck fits this space.

Re: Executing Cron Scripts Reliably at Scale

#57

Post by 8organicbits »

ohnoesjmr wrote: post #28
Scubabear68 wrote: post #21

I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.

Can you name some? I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)

There's a lot to expore here. First, I'd suggest that cron and shell scripts may not be what you want. Cron has a complex format for scheduling and can lack features like sub-minute level scheduling, maintenance windows, and other task cadences. Shell scripts are OK for small things, but often I find I move to a different language if I keep something around long enough. Do the people writing jobs know shell?

For jobs run by data analysts, airflow and python work great. For devops jobs, begrudgingly, Jenkins or GitHub Actions. But there's so many varieties.

Re: Executing Cron Scripts Reliably at Scale

#58

Post by latchkey »

Probably the simplest scheduling service I've ever used was Google Cloud Scheduler.

You give it a schedule and an endpoint (HTTP or a Cloud Task) and then it hits that endpoint on a schedule.

I go with Scheduler->Cloud Tasks->Cloud Functions, which gives you reliability and near infinite scalability.

Very easy to reason about and full monitoring of the whole stack.

Re: Executing Cron Scripts Reliably at Scale

#60

Post by teeray »

Spivak wrote: post #50
teeray wrote: post #25

Earlier quoted context omitted.

“Why are we doing this refactor in this PR along with the feature? Let’s do the feature first and then do the refactor later (read: never)”

The bet your making is you never need to do the refactor. The amount of absolutely garbage code that has never needed to change ever is easily worth a wait-and-see attitude.

It’s worth it until you need to. Then you have a nightmare on your hands because you’re never allowed to rewrite it.
Post reply on HN