Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing Cron Scripts Reliably at Scale

slack.engineering

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Executing Cron Scripts Reliably at Scale

#71

Post by wodenokoto »

AtlasBarfed wrote: post #30
WatchDog wrote: post #8

We have a slack reminder that executes on the hour, I’ve noticed that it take about 30 seconds to a minute past the hour for the reminder to actually fire.

jitter on top of the hour for massively scaled notifications is typical. OF COURSE everyone wants reminders at the even points, but what, do you scale up your system for every half-hour/hourly peak? Or just put in the TOS that the activation will jitter by a bit.

Doesn't the client actually ring the bell for the notification? They client could be told "ring on the top of the oncoming minute".

Re: Executing Cron Scripts Reliably at Scale

#72

Post by josephg »

ants_everywhere wrote: post #42
spiffytech wrote: post #40

Earlier quoted context omitted.

This line from the k8s CronJob docs has always made me nervous about adopting them: > The scheduling is approximate because there are certain circumstances where two Jobs might be created, or no Job might be created. Kubernetes tries to avoid those situations, but does not completely prevent them.

This Stack Overflow [0] answer makes it sound like that's just stating a triviality about distributed systems. For example, if the task that runs your cron is down when your cron is supposed to run, then it won't run. The slack blog says they did some tinkering like preventing nodes from going down at the top of a minute because that's when they think cron jobs are most likely to run. But at scale things are going to…

Most distributed systems at least promise they do something at least once or at most once. You can often achieve exactly once in practice with a combination of idempotent APIs and a shared database.

For a task runner, there are a lot of different behaviours you might want if the system crashes. Maybe the runner should “catch up” after coming back online. That’s easy enough to achieve if you move away from cron and track which tasks have been run in a small data store somewhere.

Re: Executing Cron Scripts Reliably at Scale

#73

Post by roydivision »

Reading this I realise an unconscious rule I've followed for years - Cron is for running jobs pertaining to that box, and that box alone. It _can_ be used to schedule anything, but that doesn't mean it should, especially business logic.

"I wrote a CRM system in ksh script, it works great!"

Re: Executing Cron Scripts Reliably at Scale

#74

Post by krisoft »

mdekkers wrote: post #69
Scubabear68 wrote: post #21

I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.

> It is the worst possible way to schedule jobs at scale. The literal evidence proves that this approach is absolutely workable at scale, so “the worst possible way” clearly doesn’t apply. The interesting question here is “how did they make this work?” The answer to which is immensely valuable to the technical community at large. “What are they building next?”, whilst interesting, is less immediately valuable as they…

> The literal evidence proves that this approach is absolutely workable at scale, so “the worst possible way” clearly doesn’t apply.

The “worst possible” to me implies that it is possible.

Think about trying to solve the same problem by hurling engineers into an erupting vulcano. It is expensive, hurts the morale, causes staff retention issues, but also fundamentally does not solve the task of scheduled task running. I would not describe that as “worst possible” because it lacks the second factor by not being a possible solution.

Re: Executing Cron Scripts Reliably at Scale

#75

Post by reacharavindh »

Isnt cron inherently easy to horizontally scale?

As in, if you have 500 cron scripts and you think you're reaching capacity of that box, just distribute the 500 scripts in one cron tab file to two boxes with 250 each?

If one cares more about the reliability of things, you can keep tab on the cron scripts starting at their times, and if they dont, then bring the box down and start a new box with the same cron tab?

Re: Executing Cron Scripts Reliably at Scale

#76

Post by blitzar »

Scubabear68 wrote: post #21

I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.

Survivorship bias.

Organisations that cobble together cron scripts for critical apps become an "an org of Slack’s size".

Companies that deploy solutions that handle all the problems with discoverability, single points of failure, failure mode options and interplanetary multi species scale when they have 0 coustomers never become "an org of Slack’s size".

Re: Executing Cron Scripts Reliably at Scale

#77

Post by ryze20245 »

I feel like a more effective way to use cron is just to dispatch jobs into a queue that will perform the actual processing. And not to do the processing within the cron scripts themselves. That way the load on the cron is light and the heavy lifting is done by your queue/worker system.

Re: Executing Cron Scripts Reliably at Scale

#78

Post by TruthWillHurt »

latchkey wrote: post #58

Probably the simplest scheduling service I've ever used was Google Cloud Scheduler. You give it a schedule and an endpoint (HTTP or a Cloud Task) and then it hits that endpoint on a schedule. I go with Scheduler->Cloud Tasks->Cloud Functions, which gives you reliability and near infinite scalability. Very easy to reason about and full monitoring of the whole stack.

o rly? what happens if they fails? can you control the backoff period? can you schedule another job in the event it fails? what if it fails before starting the function, so your in-code error handling isn't triggered? is there a "dead-letter" queue? will the next scheduled job run if the previous run failed? should it? can you define if it should or not? can you view history of executed jobs? what if your logging function that saves that history fails? can anyone add jobs or just you/superadmin? should they? and more and more.

And these are just ones I personally ran into using GCP schedulers, pub/sub and functions.

See, what you're doing is re-inventing the wheel. No matter how cool your tool is - there's always work in the edge cases beyond just running the job.

Re: Executing Cron Scripts Reliably at Scale

#79

Post by nickjj »

whorleater wrote: post #20

I would be curious why kube cron jobs didn't seem to fit the bill, my favorite part of these posts are when they have a section hinting that they explored other options picked specific tradeoffs

Kubernetes cron jobs are pretty good I must say.

I definitely don't come anywhere near Slack's scale but I've managed systems where over 3,000 cron jobs ran per day, half of which came from a cron job running every minute which usually finished in a few seconds. Some of these jobs run for X minutes too.

It's nice because there's properties you can configure for each cron job around retries and if it should be uniquely run or not. Maybe certain cron jobs should be re-tried if they fail, for others maybe it's ok to be picked up on the next interval if it fails.

Overall it's been super stable for almost 2 years which is when I started using them. Only a handful of jobs failed over this period of time and they weren't the result of Kubernetes, it was because the HTTP endpoint that was being hit from the cron job failed to respond and the cron job failure threshold was reached.

It's a good reminder that important jobs run on a schedule should be resilient to failure (saving progress, idempotent, etc.).

Post reply on HN