Slack running all their scheduled tasks on a single box using linux cron all the way up till now is an amazing point in favor of every "you don't need overly complex system architecture at your company" argument anyone has ever made.
Executing Cron Scripts Reliably at Scale
41–50 of 100 posts
Re: Executing Cron Scripts Reliably at Scale
#42Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus…
This line from the k8s CronJob docs has always made me nervous about adopting them: > The scheduling is approximate because there are certain circumstances where two Jobs might be created, or no Job might be created. Kubernetes tries to avoid those situations, but does not completely prevent them.
This Stack Overflow [0] answer makes it sound like that's just stating a triviality about distributed systems.
For example, if the task that runs your cron is down when your cron is supposed to run, then it won't run.
The slack blog says they did some tinkering like preventing nodes from going down at the top of a minute because that's when they think cron jobs are most likely to run. But at scale things are going to break when they break, and you have to weigh the pros and cons of designing the jobs to be robust to failure vs trying to organize failures to correspond to the needs of your jobs.
So I think there is space for solutions that make different tradeoffs. But it does seem vastly easier to tune an existing solution that someone else is maintaining than to build your own solution on top of Kafka.
[0] https://stackoverflow.com/questions/47691278/why-in-kubernet...
Re: Executing Cron Scripts Reliably at Scale
#43I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
Why is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.
Cron scaling? Seriously?
By definition cron does not scale, it is account by account per VM/machine with no rhyme or reason.
Re: Executing Cron Scripts Reliably at Scale
#44I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
Can you name some? I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)
I almost hate to say it, but Jenkins.
Re: Executing Cron Scripts Reliably at Scale
#45Earlier quoted context omitted.
Why is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.
Hard to believe because I have seen the cron dumpster fire at so many companies. Random cronjobs running on random boxes ends up with mystery jobs whose existence often goes unnoticed as people leave the org and docs are out of date. Random accounts with different cron jobs, with no visibility from one account to another. Box failure leads to mystery jobs failing with no way to figure out what happened. No “advanced”…
i will see your rando cron box and raise you Excel VBA macros run on Windows Task Scheduler on a spare machine in a cubicle.
Re: Executing Cron Scripts Reliably at Scale
#46[flagged]
Re: Executing Cron Scripts Reliably at Scale
#47Earlier quoted context omitted.
Hard to believe because I have seen the cron dumpster fire at so many companies. Random cronjobs running on random boxes ends up with mystery jobs whose existence often goes unnoticed as people leave the org and docs are out of date. Random accounts with different cron jobs, with no visibility from one account to another. Box failure leads to mystery jobs failing with no way to figure out what happened. No “advanced”…
i will see your rando cron box and raise you Excel VBA macros run on Windows Task Scheduler on a spare machine in a cubicle.
Never forget that the world economy is probably being held together by a handful of these.
Re: Executing Cron Scripts Reliably at Scale
#48Earlier quoted context omitted.
100% agree this is a perfect use case for Cadence or Temporal.
Coming from the "data world", how does a tool like Airflow/Dagster/Prefect differ from these?
They definitely are similar and can be used for similar functions but Cadence/Temporal are focused on code orchestration side rather than data orchestration.
Re: Executing Cron Scripts Reliably at Scale
#49this is a pretty simple cron system. curious if the authors investigated temporal and other similar workflow engines for the advanced cron feature set ( https://docs.temporal.io/workflows#spec disclaimer: i used to work there)
Wouldn’t it be a huge overkill to run your own temporal or airflow instances just to run your cron jobs? Just curious.
More overkill than writing your own system? Temporal is just plain quick to setup.
Re: Executing Cron Scripts Reliably at Scale
#50Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus…
“Why are we doing this refactor in this PR along with the feature? Let’s do the feature first and then do the refactor later (read: never)”
The bet your making is you never need to do the refactor. The amount of absolutely garbage code that has never needed to change ever is easily worth a wait-and-see attitude.