21–30 of 100 posts
Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus…
Yeah, either that or "architecting something new looks better on my resume than using an existing solution."
Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus…
I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
Why is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.
Every place I have worked cron turned into a dumpster fire.
I would be curious why kube cron jobs didn't seem to fit the bill, my favorite part of these posts are when they have a section hinting that they explored other options picked specific tradeoffs
> When designing this new, more reliable service, we decided to leverage many existing services to decrease the amount we had to build
This might explain building from scratch. Maybe the existing solutions had dependencies they didn't want to maintain and they opted for using the existing internal systems. It feels like that influenced all the rest.
I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)
Crons with precisely specified time where everyone just uses whole minutes/hours are not great practice. Very unlikely you actually need such precision in a cron job and you get spiky load.
Usual approach is to set the minute to a hash of the cron config name or something, modulo 60. Hourly jobs still run hourly, but each one on random minute.
(Setting aside how fragile that setup of avoiding pod downtime sounds)
We have a slack reminder that executes on the hour, I’ve noticed that it take about 30 seconds to a minute past the hour for the reminder to actually fire.
OF COURSE everyone wants reminders at the even points, but what, do you scale up your system for every half-hour/hourly peak? Or just put in the TOS that the activation will jitter by a bit.