S3 is Turing complete ;)
And as with everything overly flexible, people will abuse it and build over-engineered solutions on top of it, when simpler solution is already present, but possibly not obvious or poorly documented.
S3 trickery: using it as a scheduler
31–40 of 44 posts
Re: S3 trickery: using it as a scheduler
#32We're taking this serverless thing too far. We're coming up with elaborate overcomplicated schemes to accomplish simple tasks just so we can say "we're not running a server for this". Instead of doing the simple thing and running a server process that periodically checks some queue for tasks to execute, this "scheme" involves triggering a lambda function every minute, to do some IO operations on a distributed file st…
Not saying it's right or wrong solution, but saying that a simpler solution would be to take a server and put on it a job scheduler process is misleading. People shouldn't forget that it never ends with a single server and a single process, you have to manage it to make it production ready and fault tolerant.
Sometimes doing the extra step to make it serverless compatible is worth the effort.
Re: S3 trickery: using it as a scheduler
#33Dynamodb can also be used as a one-time scheduler and I will say it will look also a lot simpler than this example. In dynamodb you can set TTL for a record. When the record expires it will fire event to your designated lambda. That's it. You simply write a record, wait for the record to expire and you get notified on your lambda.
Correct me if I'm wrong, but s3 unlike dynamodb does not require a specific region, therfore it's more fault tolerance or it requires more work on dynamodb to achieve same level of stability
S3 is in regions, like Dynamo.
Re: S3 trickery: using it as a scheduler
#34Azure's service bus queue has a much better solution for this: you can schedule an item to appear in the queue at any future time. https://docs.microsoft.com/en-us/dotnet/api/microsoft.servic... Like Amazon's new SQS Lambda trigger you can then schedule a serverless function that's triggered by new items in the queue to have arbitrary scheduling of tasks. Using their Python API it's pretty nice: from datetime import…
You can do the same with AWS... - set a CloudWatch scheduled rule to schedule a lambda. - or set a rule to send an sns, subscribe a queue to the sns topic and subscribe a lambda to the queue.
CloudWatch is cron based, which means you can't schedule a one time event.
Agree regarding sns, but in our case sns == s3
Re: S3 trickery: using it as a scheduler
#35Earlier quoted context omitted.
You can do the same with AWS... - set a CloudWatch scheduled rule to schedule a lambda. - or set a rule to send an sns, subscribe a queue to the sns topic and subscribe a lambda to the queue.
CloudWatch is cron based, which means you can't schedule a one time event. Agree regarding sns, but in our case sns == s3
AWS adds a year parameter to cron. You can specify the exact time.
Why use S3? S3 events aren’t guaranteed. SNS was specifically designed for this use case.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/S...
Re: S3 trickery: using it as a scheduler
#36This isn’t using S3 as a scheduler. Cloudwatch already supports cron and rate expressions, as the post alluded to. This is a hack to schedule something once. All Cloudwatch would have to do is implement a recurrence = 1 feature but I’m guessing it's not a common enough use case.
It supports the year it should be scheduled. You could specify the exact time for it to be scheduled.
I think there is a limit on a number of scheduled events per account per region.
So when doing one-time events (by specifying a year) - the triggered lambda need to delete it.
Or have a hourly scheduled lambda to delete already fired one-time events.
Or maybe CloudWatch is so smart it can delete them by itself?
Re: S3 trickery: using it as a scheduler
#37Earlier quoted context omitted.
It supports the year it should be scheduled. You could specify the exact time for it to be scheduled.
I think there is a limit on a number of scheduled events per account per region. So when doing one-time events (by specifying a year) - the triggered lambda need to delete it. Or have a hourly scheduled lambda to delete already fired one-time events. Or maybe CloudWatch is so smart it can delete them by itself?
You would have to delete the rule. In theory, it’s just like deleting an item from a queue once it has been processed. Another advantage is that you could look in the console to see which overdue rules haven’t been deleted.
And if you really need more than 50 rules, it’s just a soft limit. You send a request to support and they will raise it.
Re: S3 trickery: using it as a scheduler
#38S3 is Turing complete ;) And as with everything overly flexible, people will abuse it and build over-engineered solutions on top of it, when simpler solution is already present, but possibly not obvious or poorly documented.
How much more obvious could it be? It’s in the Cloudwatch cron rules that the last parameter specifies the year.
Re: S3 trickery: using it as a scheduler
#39Azure's service bus queue has a much better solution for this: you can schedule an item to appear in the queue at any future time. https://docs.microsoft.com/en-us/dotnet/api/microsoft.servic... Like Amazon's new SQS Lambda trigger you can then schedule a serverless function that's triggered by new items in the queue to have arbitrary scheduling of tasks. Using their Python API it's pretty nice: from datetime import…
"you can schedule an item to appear in the queue at any future time" Nice! Anything similar in aws ?
Re: S3 trickery: using it as a scheduler
#40What is it you can achieve with this that you cannot with cron?
Multiple keywords on your resume (S3, AWS, Lambda, Serverless)
If I interviewed someone who explained that as a solution, it would definitely not help their case. The worse hire is someone who overcomplicates projects. It leads to “negative work”.