What is it you can achieve with this that you cannot with cron?
Multiple keywords on your resume (S3, AWS, Lambda, Serverless)
S3 trickery: using it as a scheduler
11–20 of 44 posts
Re: S3 trickery: using it as a scheduler
#12Azure's service bus queue has a much better solution for this: you can schedule an item to appear in the queue at any future time. https://docs.microsoft.com/en-us/dotnet/api/microsoft.servic...
Like Amazon's new SQS Lambda trigger you can then schedule a serverless function that's triggered by new items in the queue to have arbitrary scheduling of tasks.
Using their Python API it's pretty nice:
from datetime import datetime, timedelta
from azure.servicebus import ServiceBusService, Message
sbs = ServiceBusService('task-queue', shared_access_key_name=key_name, shared_access_key_value=key_value)
d = datetime.utcnow() + timedelta(minutes=1)
task = {"some": "object"}
sbs.send_queue_message(
"task-queue",
Message(
task,
broker_properties={'ScheduledEnqueueTimeUtc': d}
)
)Re: S3 trickery: using it as a scheduler
#13Dynamodb can also be used as a one-time scheduler and I will say it will look also a lot simpler than this example.
In dynamodb you can set TTL for a record. When the record expires it will fire event to your designated lambda. That's it. You simply write a record, wait for the record to expire and you get notified on your lambda.
Re: S3 trickery: using it as a scheduler
#14It's worth noting that there is currently a gotcha with S3 Event Notifications such that they are not guaranteed. As a result, you may end up missing out on events.
Is there an open issue or documentation for this somewhere? We rely quite heavily on lambdas trigger by S3 events and have never experienced this
Re: S3 trickery: using it as a scheduler
#15Wait what? You can invoke lambda functions directly via the invoke api. No need to use the s3 api. Am I missing something?
Re: S3 trickery: using it as a scheduler
#16We're taking this serverless thing too far. We're coming up with elaborate overcomplicated schemes to accomplish simple tasks just so we can say "we're not running a server for this". Instead of doing the simple thing and running a server process that periodically checks some queue for tasks to execute, this "scheme" involves triggering a lambda function every minute, to do some IO operations on a distributed file st…
Agreed, lol. I’d say let Tech Darwinism take its toll on the weak.
Re: S3 trickery: using it as a scheduler
#17It's worth noting that there is currently a gotcha with S3 Event Notifications such that they are not guaranteed. As a result, you may end up missing out on events.
Is there an open issue or documentation for this somewhere? We rely quite heavily on lambdas trigger by S3 events and have never experienced this
> an open issue
If only; this is AWS. You pretty much need premium support just to tell them their products are broken. Before someone says "forums", the forums are a joke.
Re: S3 trickery: using it as a scheduler
#18CloudWatch Events already supports triggering Lambda functions on a cron schedule. You don't even need S3. This seems like a ton of work to basically invoke a function every n minutes.
Re: S3 trickery: using it as a scheduler
#19I was expecting this to be about combining Lambda with S3 lifecycle management to make a poor man’s cron.
Now I kinda want to try that myself.
Re: S3 trickery: using it as a scheduler
#20CloudWatch Events already supports triggering Lambda functions on a cron schedule. You don't even need S3. This seems like a ton of work to basically invoke a function every n minutes.
If you want a reoccurring task then you are right, but if you want to run a one time task in a specific time in the future depending on internal logic, then it won't work