Live data from Hacker News

Executing Cron Scripts Reliably at Scale

slack.engineering

31–40 of 100 posts

Re: Executing Cron Scripts Reliably at Scale

#31

Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus…

> "1 box running shell scripts with flock" to "mega custom thing"

Literally went from a working "tidy little house" to building a massive sky-scraper.

Obviously Slack operates at a much larger scale than I do...but holy moly.

hey, good luck to them with that!

Re: Executing Cron Scripts Reliably at Scale

#32
post #23

Earlier quoted context omitted.

Why is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.

Hard to believe because I have seen the cron dumpster fire at so many companies. Random cronjobs running on random boxes ends up with mystery jobs whose existence often goes unnoticed as people leave the org and docs are out of date. Random accounts with different cron jobs, with no visibility from one account to another. Box failure leads to mystery jobs failing with no way to figure out what happened. No “advanced”…

Or you could have a canonical cron box.

Re: Executing Cron Scripts Reliably at Scale

#33

Earlier quoted context omitted.

100% agree this is a perfect use case for Cadence or Temporal.

Coming from the "data world", how does a tool like Airflow/Dagster/Prefect differ from these?

It doesn't. They didn't look for it. This is exactly what DAG based workflow systems to, in a modern professional manner. Crons don't do a DAG.

Re: Executing Cron Scripts Reliably at Scale

#34
> we implemented logic to prevent the node from going down at the top of a minute when possible since — given the nature of cron — that is when it is likely that scripts will need to be scheduled to run

Why not smear the start time of the jobs across seconds of that minute to avoid any thundering herd problems? How much functionality relies on a script being invoked at exactly the :00 mark? And if the functionality depends on that exact timing, doesn’t it suggest something is fragile and could be redesigned to be more resilient?

Re: Executing Cron Scripts Reliably at Scale

#35

Earlier quoted context omitted.

Hard to believe because I have seen the cron dumpster fire at so many companies. Random cronjobs running on random boxes ends up with mystery jobs whose existence often goes unnoticed as people leave the org and docs are out of date. Random accounts with different cron jobs, with no visibility from one account to another. Box failure leads to mystery jobs failing with no way to figure out what happened. No “advanced”…

Or you could have a canonical cron box.

Or a fleet? Some of the problems up-thread feel organizational rather than tooling.

Re: Executing Cron Scripts Reliably at Scale

#36

I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.

Can you name some? I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)

Build the UI or API for the report, so you can ad-hoc from there. Cron uses that interface. You don't custom the cron, you cron the custom.

Re: Executing Cron Scripts Reliably at Scale

#37

> we implemented logic to prevent the node from going down at the top of a minute when possible since — given the nature of cron — that is when it is likely that scripts will need to be scheduled to run Why not smear the start time of the jobs across seconds of that minute to avoid any thundering herd problems? How much functionality relies on a script being invoked at exactly the :00 mark? And if the functionality d…

At their scale, staggering script start times over a 60 second window likely wouldn’t have much of an impact if they are experiencing a thundering herd, imo. If it did help, it would be a bandaid and ticking time bomb before someone has to actually solve the load problem that staggering start times kicked down the road

Re: Executing Cron Scripts Reliably at Scale

#38

Earlier quoted context omitted.

100% agree this is a perfect use case for Cadence or Temporal.

Coming from the "data world", how does a tool like Airflow/Dagster/Prefect differ from these?

I recently evaluated Dagster, Prefect, and Flyte for a data pipeliney workflow and ended up going with Temporal.

The shared feature between Temporal and those three is the workflow orchestration piece. All 3 can manage a dependency graph of jobs, handle retries, start from checkpoints, etc.

At a high level the big reason they’re different is Temporal is entirely focused on the orchestration piece, and the others are much more focused on the data piece, which comes out in a lot of the different features. Temporal has SDKs in most languages, and has a queuing system that allows you to run different workflows or even activities (tasks within a workflow) in different workers, manage concurrency, etc. You can write a parent workflow that orchestrates sub-workflows that could live in 5 other services. It’s just really composable and fits much more nicely into the critical path of your app.

Prefect is probably the closest of your list to temporal, in that it’s less opinionated than others about the workflows being “data oriented”, but it’s still only in python, and it deosn't have queueing. In short this means that your workflows are kinda supposed to run in one box running python somewhere. Temporal will let you define a 10 part workflow where two parts run on a python service running with a GPU, and the remaining parts are running in the same node.js process as your main server.

Dagster’s feature set is even more focused on data-workflows, as your workflows are meant to produce data “assets” which can be materialized/cached, etc.

They’re pretty much all designed for a data engineering team to manage many individual pipelines that are external from your application code, whereas temporal is designed to be a system that manages workflow complexity for code that (more often) runs in your application.

Re: Executing Cron Scripts Reliably at Scale

#39
post #23

I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.

Why is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.

It’s no longer simple when you have platform code to prevent nodes from disappearing or dying on the minute on a kubernetes cluster the size of slack. A triggered pulse event stream would have done the trick to invoke a lambda or call code for every “thing” that needed a beat. Kubernetes comes with a scheduler…

Re: Executing Cron Scripts Reliably at Scale

#40

Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus…

This line from the k8s CronJob docs has always made me nervous about adopting them:

> The scheduling is approximate because there are certain circumstances where two Jobs might be created, or no Job might be created. Kubernetes tries to avoid those situations, but does not completely prevent them.

Post reply on HN