Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deno Cron

deno.com

Post reply on HN
Print view View original

51–60 of 190 posts

Re: Deno Cron

#51

Post by SheinhardtWigCo »

igorzi wrote: post #46
nwjsmith wrote: post #30

Earlier quoted context omitted.

Is the spikiness of cron schedules going to cause you operational problems? You're going to end up with a _lot_ of jobs scheduled at "0 0 * * *".

Yes, we're anticipating more spiky workloads because of this. Deno Deploy is already designed to handle spikes, but we also have a few additional mitigations in place for Cron. For example, we will limit concurrent dispatches for the same project/user/organization, which may slightly delay the execution of specific cron tasks.

Charge 25% more for every 0 in the scheduling expression, problem solved :-)

Re: Deno Cron

#52

Post by notpachet »

klabb3 wrote: post #49
vvpan wrote: post #37

I am a lead on a small startup team and one of the biggest pain points is dealing with infrastructure. We have no dedicated devops person and much of that work falls on me and other people who would be better writing code. I think the cloud paradigm is experiencing a shift. Few of us want to deal with cloud infrastructure (whether clicking around or via Terraform or equivalent) to execute a function every X minutes -…

> 1. Services like this cron with Dyno Deploy or Vercel where the cloud things are abstracted away for you. I agree with the sentiment, but to me it’s a shame this even qualifies as abstraction in the first place. Periodic execution of a function is extremely basic and already exists in almost every programming environment. If this needs to be a service (instead of a library call), there is something deeply wrong els…

I wonder why more people don't just set up a cheap VM and run this sort of thing via crontab.

Re: Deno Cron

#53

Post by jacobyoder »

mike_d wrote: post #43
vvpan wrote: post #37

I am a lead on a small startup team and one of the biggest pain points is dealing with infrastructure. We have no dedicated devops person and much of that work falls on me and other people who would be better writing code. I think the cloud paradigm is experiencing a shift. Few of us want to deal with cloud infrastructure (whether clicking around or via Terraform or equivalent) to execute a function every X minutes -…

It is really disappointing to me when I hear takes like this. As an industry we are splitting the roles of programmers and engineers so that programmers can just throw spaghetti at the wall and it is the engineers problem to happily run it. We need more well rounded people that also fundamentally understand how their code is executing on the backend so performance and cost can be optimized.

> We need more well rounded people that also fundamentally understand how their code is executing on the backend so performance and cost can be optimized.

There's also some sort of balance that needs to be struck. As a 'web developer'... to be 'well balanced', I need to understand (or dig in to) the minutia of:

* JS build tools, syntax, oddities, versions, and be able to troubleshoot these in a variety of environment (works locally with these architectures, and runs on production, but the CI pipeline changed to accommodate someone else and now I have to unstuck all this).

* SQL - I need to understand the implications and nuance of various indexing strategies because... hey, PG 14 changed and now our caching approach needs tuning. Should I be using b-tree indices, or something else? Should my indices be clustered? How do I manage replication and backups? And I need to understand the tradeoffs of ORMs vs hand-rolled, vs stored procedures, time-series databases, performance impact of views and materialization. And security.

* Application level - I need to understand various aspects of security at the application layer, and balance security vs usability vs accessibility vs performance vs feedback from various program/product managers and end users. Oh, and I need to be able to troubleshoot the application in a variety of contexts. Oh... and mobile. I need to be conversant and 'well rounded' in various flavors of mobile development (hybrid vs native vs whatever).

* System updates - what? I'm still on JDK 11? WTF? Why aren't you on the current versions? We need to update pronto, because there's too many security issues we'll get dinged on!

* UI - I need to be a CSS master, and if I'm not already on the tailwind bandwagon, I risk losing my job. Or... at least, I'll fall behind as I try to incorporate the new tailwind UI work the new person did because they couldn't be bothered to learn bootstrap 5. And... accessibility - need to be an expert in that.

* Deployment - hey, I now need to be able to understand multiple deployment approaches and processes, be fluent in docker/k8s and various tools, be able to troubleshoot these self-sufficiently. Doesn't matter if these are necessarily the right tools for the situation - someone else dictated this and lobbying for anything else means you're afraid to learn new things.

And... I should be cheery and helpful and positive about all of this being my responsibility. Because when I ask for folks from another team to help, I get told I need to 'own' the project, and I can't just 'throw it over the wall' and expect someone else to do something for me. If I push back on anything, I risk getting labelled "old" and "a dinosaur" and "afraid of change" and "unwilling to learn new things".

tldr - if you actually have teams of people with related yet diverse skills, consider letting each play to their strengths, help support them in their strengths, and organize around everyone doing what they're best at. The 'well rounded' thing has its limits.

Re: Deno Cron

#54

Post by mgkimsal »

koolba wrote: post #39

> How exactly does Deno Deploy know there’s a cron in your code, even when there’s no web server handling requests? > When a new production deployment of your project is created, an ephemeral V8 isolate is used to evaluate your project’s top-level scope and to discover any Deno.cron definitions. A global cron scheduler is then updated with your project’s latest cron definitions, which includes updates to your existin…

Where better to put cron work? In another repo? If the cron jobs are 100% unrelated to the codebase, yeah, it might be weird. If the jobs are "run the foo() method every 2am"... having that info colocated with the foo() method itself makes a lot of sense, imo.

Re: Deno Cron

#55

Post by aleksiy123 »

danfarrelly wrote: post #27
aleksiy123 wrote: post #16

Nice, I was just wondering if there was an opportunity for a product similar to vercel for batch jobs. This isn't quite that but close. From my experience there 5 main components for larger scale system. - Request server - database - queue - background worker - offline batch jobs I feel like so far serverless has the first 3 fairly well developed but the last two are still underdeveloped. But maybe I'm just not aware…

This is pretty much what Inngest is ( https://www.inngest.com/ ). Runs on Deno as well.

Ingest is definitely nice. Its design is an orchestrator/scheduler where you offload your workers as serverless functions.

The only issue is for background jobs you need to design it in such way to not run in to timeouts.

Which makes it slightly more complex then just having a single executable running periodically.

There is also https://www.defer.run/ which run your code on their infra and don't have timeouts. But they only support node/bun at the moment.

Re: Deno Cron

#56

Post by everforward »

klabb3 wrote: post #49
vvpan wrote: post #37

I am a lead on a small startup team and one of the biggest pain points is dealing with infrastructure. We have no dedicated devops person and much of that work falls on me and other people who would be better writing code. I think the cloud paradigm is experiencing a shift. Few of us want to deal with cloud infrastructure (whether clicking around or via Terraform or equivalent) to execute a function every X minutes -…

> 1. Services like this cron with Dyno Deploy or Vercel where the cloud things are abstracted away for you. I agree with the sentiment, but to me it’s a shame this even qualifies as abstraction in the first place. Periodic execution of a function is extremely basic and already exists in almost every programming environment. If this needs to be a service (instead of a library call), there is something deeply wrong els…

> Periodic execution of a function is extremely basic and already exists in almost every programming environment.

Periodic execution is surprisingly complicated. What happens when your node is down when it was supposed to execute and then comes back up? What happens around timezone changes and leap years? What does this thing do if a run takes so long that the previous run is still going? How do you get logs/notifications if it fails?

"Best effort" periodic execution is super easy, where you try to make a thing happen every period but it's okay if it doesn't happen as long as it does happen at some point in the near future (i.e. periodic clean up jobs, or maybe batch emails).

There's also a salient argument that you shouldn't run things inside the same process/VM if you can avoid it. It'd be a real shame if the production app went down because the cleanup cronjob went haywire and choked out the CPU.

Re: Deno Cron

#57

Post by seqizz »

vvpan wrote: post #37

I am a lead on a small startup team and one of the biggest pain points is dealing with infrastructure. We have no dedicated devops person and much of that work falls on me and other people who would be better writing code. I think the cloud paradigm is experiencing a shift. Few of us want to deal with cloud infrastructure (whether clicking around or via Terraform or equivalent) to execute a function every X minutes -…

> one of the biggest pain points is dealing with infrastructure

> We have no dedicated devops person

I understand that not every startup can afford an extra person for infra-heavy tasks, but this should not be surprising. Any tool which looks like an abstraction to you might get complex by time, and steal your time. The cloud was supposed to fix some of these, look where we are now.

Re: Deno Cron

#58

Post by aleksiy123 »

slig wrote: post #42
aleksiy123 wrote: post #16

Nice, I was just wondering if there was an opportunity for a product similar to vercel for batch jobs. This isn't quite that but close. From my experience there 5 main components for larger scale system. - Request server - database - queue - background worker - offline batch jobs I feel like so far serverless has the first 3 fairly well developed but the last two are still underdeveloped. But maybe I'm just not aware…

You should check Windmill.dev.

I will check out windmill. Though it does look somewhat complex.

One tangentially related is I'm looking for somewhere to just run a python script once a day.

I'll see if I can do that with windmill.

Re: Deno Cron

#60

Post by twic »

everforward wrote: post #56
klabb3 wrote: post #49

Earlier quoted context omitted.

> 1. Services like this cron with Dyno Deploy or Vercel where the cloud things are abstracted away for you. I agree with the sentiment, but to me it’s a shame this even qualifies as abstraction in the first place. Periodic execution of a function is extremely basic and already exists in almost every programming environment. If this needs to be a service (instead of a library call), there is something deeply wrong els…

> Periodic execution of a function is extremely basic and already exists in almost every programming environment. Periodic execution is surprisingly complicated. What happens when your node is down when it was supposed to execute and then comes back up? What happens around timezone changes and leap years? What does this thing do if a run takes so long that the previous run is still going? How do you get logs/notifica…

I am a big fan of avoiding service proliferation and cloud complexity, and just making really good use of the facilities provided by your language and OS of choice.

But for exactly the reasons you outline, scheduling jobs is one of the few things where i would be eager to use some reliable, observable, carefully-engineered service.

Post reply on HN