S3 trickery: using it as a scheduler
hackernoon.com
S3 trickery: using it as a scheduler
1–10 of 44 posts
Re: S3 trickery: using it as a scheduler
#2It's worth noting that there is currently a gotcha with S3 Event Notifications such that they are not guaranteed. As a result, you may end up missing out on events.
Re: S3 trickery: using it as a scheduler
#3It's worth noting that there is currently a gotcha with S3 Event Notifications such that they are not guaranteed. As a result, you may end up missing out on events.
Can you elaborate more? Is this related to S3 uptime SLA or is there a different reason?
Re: S3 trickery: using it as a scheduler
#4It's worth noting that there is currently a gotcha with S3 Event Notifications such that they are not guaranteed. As a result, you may end up missing out on events.
Can you elaborate more? Is this related to S3 uptime SLA or is there a different reason?
I believe it is different to the S3 uptime SLA.
You can measure the impact and potentially automate recovery of missed events by:
1) Keep a track of events published.
2) Generate an S3 inventory daily.
3) Compare events received to objects listed in inventory.
You rarely end up with fewer events than objects, but it does occur.
I've personally only observed this with SNS target, but due to it being a problem with S3, I believe Lambda can fall afoul of this too.
Re: S3 trickery: using it as a scheduler
#5What is it you can achieve with this that you cannot with cron?
Re: S3 trickery: using it as a scheduler
#6It's worth noting that there is currently a gotcha with S3 Event Notifications such that they are not guaranteed. As a result, you may end up missing out on events.
S3 also doesn't provide a linearizable consistency model or even a vague approximation of one. You can't rely on the events you try to schedule happening in the order you try to schedule them in, or even happening at all.
This seems overcomplicated compared to using a regular timed event to trigger a lambda and having it decide what to execute conditionally.
Re: S3 trickery: using it as a scheduler
#7This is misleading.
The title should be "Using s3 as storage and some side thingy that runs every minute as a scheduler".
Re: S3 trickery: using it as a scheduler
#8It's worth noting that there is currently a gotcha with S3 Event Notifications such that they are not guaranteed. As a result, you may end up missing out on events.
S3 also doesn't provide a linearizable consistency model or even a vague approximation of one. You can't rely on the events you try to schedule happening in the order you try to schedule them in, or even happening at all. This seems overcomplicated compared to using a regular timed event to trigger a lambda and having it decide what to execute conditionally.
I wasn't really too worried about picking apart the approach. If order mattered, you'd be looking at other approaches anyway.
Mainly just calling out a gotcha where you might quietly miss out on scheduled events with no warning.
For example:
1. Object written to successfully to bucket.
2. `s3:ObjectCreated:Put` is _never_ delivered.
The possibility of duplicate events are warned about a lot in the AWS ecosystem, and this sets up an expectation of "at-least-once" delivery.
Re: S3 trickery: using it as a scheduler
#9We're taking this serverless thing too far. We're coming up with elaborate overcomplicated schemes to accomplish simple tasks just so we can say "we're not running a server for this".
Instead of doing the simple thing and running a server process that periodically checks some queue for tasks to execute, this "scheme" involves triggering a lambda function every minute, to do some IO operations on a distributed file store where the execution date metadata is embedded in the filenames to see if there is some task to execute. If there is, some more distributed IO operations are required to move the file to another part of the distributed file storage, where some listener will notice a new file was added and trigger the actual lambda function with the task-specific code...
But hey, you're not running any servers yourself...
Re: S3 trickery: using it as a scheduler
#10> What happens when you want to schedule a one-time event? You are stuck.
Or you create a rule for a specific date / time. Any reason why that would not work?