Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 trickery: using it as a scheduler

hackernoon.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: S3 trickery: using it as a scheduler

#21

Post by efi_mk »

_pdp_ wrote: post #13

Dynamodb can also be used as a one-time scheduler and I will say it will look also a lot simpler than this example. In dynamodb you can set TTL for a record. When the record expires it will fire event to your designated lambda. That's it. You simply write a record, wait for the record to expire and you get notified on your lambda.

Correct me if I'm wrong, but s3 unlike dynamodb does not require a specific region, therfore it's more fault tolerance or it requires more work on dynamodb to achieve same level of stability

Re: S3 trickery: using it as a scheduler

#22

Post by jimmytucson »

This isn’t using S3 as a scheduler. Cloudwatch already supports cron and rate expressions, as the post alluded to. This is a hack to schedule something once.

All Cloudwatch would have to do is implement a recurrence = 1 feature but I’m guessing it's not a common enough use case.

Re: S3 trickery: using it as a scheduler

#24

Post by k__ »

efi_mk wrote: post #20
cddotdotslash wrote: post #18

CloudWatch Events already supports triggering Lambda functions on a cron schedule. You don't even need S3. This seems like a ton of work to basically invoke a function every n minutes.

If you want a reoccurring task then you are right, but if you want to run a one time task in a specific time in the future depending on internal logic, then it won't work

Isn't this a job for Step Functions?

Re: S3 trickery: using it as a scheduler

#25

Post by Serow225 »

mcintyre1994 wrote: post #12

Azure's service bus queue has a much better solution for this: you can schedule an item to appear in the queue at any future time. https://docs.microsoft.com/en-us/dotnet/api/microsoft.servic... Like Amazon's new SQS Lambda trigger you can then schedule a serverless function that's triggered by new items in the queue to have arbitrary scheduling of tasks. Using their Python API it's pretty nice: from datetime import…

Thanks for the info, tucking that away! :)

Re: S3 trickery: using it as a scheduler

#26

Post by scarface74 »

vbsteven wrote: post #9

We're taking this serverless thing too far. We're coming up with elaborate overcomplicated schemes to accomplish simple tasks just so we can say "we're not running a server for this". Instead of doing the simple thing and running a server process that periodically checks some queue for tasks to execute, this "scheme" involves triggering a lambda function every minute, to do some IO operations on a distributed file st…

Don’t blame Serverless. Blame it on a poor implementation. AWS Cloudwatch supports cron rules that include the year that it should be scheduled and it can trigger lambdas directly. There was no reason he had to over complicate this.

As far as using a queue, lambda also supports triggering based on a queue.

Re: S3 trickery: using it as a scheduler

#27

Post by scarface74 »

jimmytucson wrote: post #22

This isn’t using S3 as a scheduler. Cloudwatch already supports cron and rate expressions, as the post alluded to. This is a hack to schedule something once. All Cloudwatch would have to do is implement a recurrence = 1 feature but I’m guessing it's not a common enough use case.

It supports the year it should be scheduled. You could specify the exact time for it to be scheduled.

Re: S3 trickery: using it as a scheduler

#28

Post by efi_mk »

k__ wrote: post #24
efi_mk wrote: post #20

Earlier quoted context omitted.

If you want a reoccurring task then you are right, but if you want to run a one time task in a specific time in the future depending on internal logic, then it won't work

Isn't this a job for Step Functions?

Step function is also a good idea

Re: S3 trickery: using it as a scheduler

#29

Post by scarface74 »

mcintyre1994 wrote: post #12

Azure's service bus queue has a much better solution for this: you can schedule an item to appear in the queue at any future time. https://docs.microsoft.com/en-us/dotnet/api/microsoft.servic... Like Amazon's new SQS Lambda trigger you can then schedule a serverless function that's triggered by new items in the queue to have arbitrary scheduling of tasks. Using their Python API it's pretty nice: from datetime import…

You can do the same with AWS...

- set a CloudWatch scheduled rule to schedule a lambda.

- or set a rule to send an sns, subscribe a queue to the sns topic and subscribe a lambda to the queue.

Re: S3 trickery: using it as a scheduler

#30

Post by scarface74 »

efi_mk wrote: post #20
cddotdotslash wrote: post #18

CloudWatch Events already supports triggering Lambda functions on a cron schedule. You don't even need S3. This seems like a ton of work to basically invoke a function every n minutes.

If you want a reoccurring task then you are right, but if you want to run a one time task in a specific time in the future depending on internal logic, then it won't work

You add the year to the cron job...

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/S...

Post reply on HN