I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
Can you name some? I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)
this is a pretty simple cron system. curious if the authors investigated temporal and other similar workflow engines for the advanced cron feature set ( https://docs.temporal.io/workflows#spec disclaimer: i used to work there)
Wouldn’t it be a huge overkill to run your own temporal or airflow instances just to run your cron jobs? Just curious.
Probably the simplest scheduling service I've ever used was Google Cloud Scheduler. You give it a schedule and an endpoint (HTTP or a Cloud Task) and then it hits that endpoint on a schedule. I go with Scheduler->Cloud Tasks->Cloud Functions, which gives you reliability and near infinite scalability. Very easy to reason about and full monitoring of the whole stack.
o rly? what happens if they fails? can you control the backoff period? can you schedule another job in the event it fails? what if it fails before starting the function, so your in-code error handling isn't triggered? is there a "dead-letter" queue? will the next scheduled job run if the previous run failed? should it? can you define if it should or not? can you view history of executed jobs? what if your logging fun…
As the person below states, you sound bitter.
There is nothing about your questions that wouldn't apply to any system.
Literally all of your questions are answered in the rather well written documentation. I could go through them and answer them for you, but I don't think you'd really appreciate that.
Kubernetes cron jobs are pretty good I must say. I definitely don't come anywhere near Slack's scale but I've managed systems where over 3,000 cron jobs ran per day, half of which came from a cron job running every minute which usually finished in a few seconds. Some of these jobs run for X minutes too. It's nice because there's properties you can configure for each cron job around retries and if it should be uniquel…
Do you capture all of your job code in a single image and reference execution paths on container startup per job? Or, are you building an image per job?
The jobs all run curl commands to a specific API endpoint with a specific bearer token. Those tokens are loaded through an env through SealedSecrets.
They all use the public curl image where I override the command in the Kubernetes cron job definition. The job container itself starts almost instantly since there's no app to boot.
If I had a case you're describing I would use the main app's image and run a specific command, in this case I'm assuming if there's not an API endpoint it would be some callable script that lives in your app's code / image.
Yeah, either that or "architecting something new looks better on my resume than using an existing solution."
Do you have "implemented a task scheduler" on your resume?
Personally I'd see it as a negative vs using an industry standard solution.
However, I'm sure some folks would be tempted to add something like "designed and implemented a distributed task scheduler and execution engine for generalized asynchronous jobs utilized by X number of devs across Y teams" to their resumes.
Do you have "implemented a task scheduler" on your resume?
Personally I'd see it as a negative vs using an industry standard solution. However, I'm sure some folks would be tempted to add something like "designed and implemented a distributed task scheduler and execution engine for generalized asynchronous jobs utilized by X number of devs across Y teams" to their resumes.
My question is on the line of do you think it's relevant enough so that it would deserve being added to a resume? Even if it's something you personally don't like.
Because I can't imagine why it would award that relevance. It's right there with "implemented function to reverse a list because the stdlib had a bug".
this is a pretty simple cron system. curious if the authors investigated temporal and other similar workflow engines for the advanced cron feature set ( https://docs.temporal.io/workflows#spec disclaimer: i used to work there)
Wouldn’t it be a huge overkill to run your own temporal or airflow instances just to run your cron jobs? Just curious.
There are certainly use cases for which it's more than is required. Like the most simple would be adding a cron string to a GitHub Action or Vercel function, but in most cases, and certainly Slack's case, you want more reliability, scalability, flexibility, and/or observability. And Temporal has extreme levels of those things, including pausing & editing schedules and seeing the current status of all triggered scripts/functions and all the steps a function has taken so far, and guaranteeing the triggered function completes, including ensuring that if the process or container dies, the function continues running on another one. Even if you don't care about all those things, you might care about some of them in the future, and it doesn't hurt to run a system that has capabilities you don't use.
They definitely are similar and can be used for similar functions but Cadence/Temporal are focused on code orchestration side rather than data orchestration.
I find that comparison interesting, because I don't think of Airflow as particularly data-oriented in terms of its features and core functionality. I tend to think of Airflow as "Cron + Make", with any "data-oriented" features being nice to have, but not essential. I'm substantially less familiar with Dagster and Prefect so can't comment as much on those. Maybe the most data-oriented thing about Airflow is its concep…
I’ve only recently encountered airflow for the first time, and have been surprised at how half-baked it is for being damn near the industry standard (as far as open source, anyway). And it was a lot worse until recently!
Dynamic task dispatch being a relatively recent feature. The fundamental design imposing lots of structure (well, kind of—you can skip lots of it, but it takes time to figure that out) to practically no benefit (and god, is the terminology dumb, made all the more so because half the stuff it names is nearly useless). “Oh yeah the scheduler just crashes or locks up while still health-checking all the time, standard practice to so restart it frequently” posted on a hundred different issues dating from yesterday to years ago (many fixed! And yet…). It’s pretty bad at passing data between tasks (see again: lots of structure, little benefit)
This works great in my experience. A lifetime ago I scaled up cron jobs for a client with Gearman. Using cron to trigger jobs on the Gearman server and the pool of runners to do all the work. This proved to be so reliable they still use the system today, over 10 years later.
Really cool! For the Gearman workers, did you load jobs dynamically? Or, would you have to re-deploy for new jobs/updated jobs?
As I recall, all the jobs are checked into a repo that is deployed to all the runners, which each start gearman workers for their assigned role.