Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executing Cron Scripts Reliably at Scale

slack.engineering

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Executing Cron Scripts Reliably at Scale

#81

Post by 1a527dd5 »

TruthWillHurt wrote: post #78
latchkey wrote: post #58

Probably the simplest scheduling service I've ever used was Google Cloud Scheduler. You give it a schedule and an endpoint (HTTP or a Cloud Task) and then it hits that endpoint on a schedule. I go with Scheduler->Cloud Tasks->Cloud Functions, which gives you reliability and near infinite scalability. Very easy to reason about and full monitoring of the whole stack.

o rly? what happens if they fails? can you control the backoff period? can you schedule another job in the event it fails? what if it fails before starting the function, so your in-code error handling isn't triggered? is there a "dead-letter" queue? will the next scheduled job run if the previous run failed? should it? can you define if it should or not? can you view history of executed jobs? what if your logging fun…

Wow, you sound a little bitter.

We use AWS CloudWatch Events at work, and it's fine.

Re: Executing Cron Scripts Reliably at Scale

#82

Post by chronid »

getoffmyyawn wrote: post #80

That there are numerous mature battle tested open source solutions to distributed and/or centrally managed job queues that it really makes me wonder how they justified building something from scratch.

I think there's a bit of "they could" but also something that is considered very little in many contexts unless you have experienced the contrary: integration is costly and integrating properly sometimes is more work than doing something from "scratch", so you don't do it and then you have a mess that hurts you in the long run.

Re: Executing Cron Scripts Reliably at Scale

#83

Post by neomantra »

nicksaroha wrote: post #17
swyx wrote: post #5

this is a pretty simple cron system. curious if the authors investigated temporal and other similar workflow engines for the advanced cron feature set ( https://docs.temporal.io/workflows#spec disclaimer: i used to work there)

Wouldn’t it be a huge overkill to run your own temporal or airflow instances just to run your cron jobs? Just curious.

Depends on how many cron jobs you have and what you need out of it?

Operating Temporal is not that hard -- you can start with `temporal --dev` on your own box. I have a "Nomad-Temporal" Terraform Module to stand one up on Nomad. [1] Temporal has Helm Charts for Kubernetes [2]. There is also Temporal Cloud [3].

That said, there is currently a chasm between "script in cronjob" to "scheduled task in Temporal". The focus of Temporal is more "Enterprise, get your Business Processes on Temporal", not "soloist, ditch your cron".

There's certainly space for somebody to a make DAG dataflow thing or lower-code product over Temporal. Airplane.dev [4] was built on Temporal and was approaching this; acquired by AirTable.

[1] https://github.com/neomantra/terraform-nomad-temporal [2] https://github.com/temporalio/helm-charts [3] https://temporal.io/cloud [4] https://www.airplane.dev

Re: Executing Cron Scripts Reliably at Scale

#84

Post by getoffmyyawn »

chronid wrote: post #82
getoffmyyawn wrote: post #80

That there are numerous mature battle tested open source solutions to distributed and/or centrally managed job queues that it really makes me wonder how they justified building something from scratch.

I think there's a bit of "they could" but also something that is considered very little in many contexts unless you have experienced the contrary: integration is costly and integrating properly sometimes is more work than doing something from "scratch", so you don't do it and then you have a mess that hurts you in the long run.

I'm sure it's indeed something like that. I think it also comes down to, at least partly, having a culture that is more about building components than systems. I suspect it could also be the "buzz" factor. The press release about building a new system always seems more exciting than one about solving a familiar problem with boring old existing software.

Re: Executing Cron Scripts Reliably at Scale

#85

Post by getoffmyyawn »

ryze20245 wrote: post #77

I feel like a more effective way to use cron is just to dispatch jobs into a queue that will perform the actual processing. And not to do the processing within the cron scripts themselves. That way the load on the cron is light and the heavy lifting is done by your queue/worker system.

This works great in my experience.

A lifetime ago I scaled up cron jobs for a client with Gearman. Using cron to trigger jobs on the Gearman server and the pool of runners to do all the work. This proved to be so reliable they still use the system today, over 10 years later.

Re: Executing Cron Scripts Reliably at Scale

#86

Post by marcosdumay »

spiffytech wrote: post #40
xyzzy123 wrote: post #11

Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus…

This line from the k8s CronJob docs has always made me nervous about adopting them: > The scheduling is approximate because there are certain circumstances where two Jobs might be created, or no Job might be created. Kubernetes tries to avoid those situations, but does not completely prevent them.

Looks like they made a cron on top of an eventual-consistent database.

Yeah, I'd avoid that too.

Re: Executing Cron Scripts Reliably at Scale

#87

Post by marcosdumay »

noptd wrote: post #16
xyzzy123 wrote: post #11

Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus…

Yeah, either that or "architecting something new looks better on my resume than using an existing solution."

Do you have "implemented a task scheduler" on your resume?

Re: Executing Cron Scripts Reliably at Scale

#88

Post by nerdponx »

jaydeegee wrote: post #48
nerdponx wrote: post #14

Earlier quoted context omitted.

Coming from the "data world", how does a tool like Airflow/Dagster/Prefect differ from these?

They definitely are similar and can be used for similar functions but Cadence/Temporal are focused on code orchestration side rather than data orchestration.

I find that comparison interesting, because I don't think of Airflow as particularly data-oriented in terms of its features and core functionality. I tend to think of Airflow as "Cron + Make", with any "data-oriented" features being nice to have, but not essential.

I'm substantially less familiar with Dagster and Prefect so can't comment as much on those.

Maybe the most data-oriented thing about Airflow is its concept of a data interval, where each DAG run is associated with some "logical date" and an interval of time that starts from the logical date (inclusive) and ends at the next logical date in the schedule (exclusive). The idea is that if you have a daily task that runs at 1 AM, then the task is expected to operate on data starting from "yesterday at 1 AM" until "today at 1 AM". But it's entirely up to the user/developer what you actually do with those logical date ranges, and you're free to ignore them entirely if you don't need them.

Re: Executing Cron Scripts Reliably at Scale

#89

Post by djboz »

nickjj wrote: post #79
whorleater wrote: post #20

I would be curious why kube cron jobs didn't seem to fit the bill, my favorite part of these posts are when they have a section hinting that they explored other options picked specific tradeoffs

Kubernetes cron jobs are pretty good I must say. I definitely don't come anywhere near Slack's scale but I've managed systems where over 3,000 cron jobs ran per day, half of which came from a cron job running every minute which usually finished in a few seconds. Some of these jobs run for X minutes too. It's nice because there's properties you can configure for each cron job around retries and if it should be uniquel…

Do you capture all of your job code in a single image and reference execution paths on container startup per job? Or, are you building an image per job?

Re: Executing Cron Scripts Reliably at Scale

#90

Post by djboz »

getoffmyyawn wrote: post #85
ryze20245 wrote: post #77

I feel like a more effective way to use cron is just to dispatch jobs into a queue that will perform the actual processing. And not to do the processing within the cron scripts themselves. That way the load on the cron is light and the heavy lifting is done by your queue/worker system.

This works great in my experience. A lifetime ago I scaled up cron jobs for a client with Gearman. Using cron to trigger jobs on the Gearman server and the pool of runners to do all the work. This proved to be so reliable they still use the system today, over 10 years later.

Really cool! For the Gearman workers, did you load jobs dynamically? Or, would you have to re-deploy for new jobs/updated jobs?
Post reply on HN