41–44 of 44 posts
S3 is Turing complete ;) And as with everything overly flexible, people will abuse it and build over-engineered solutions on top of it, when simpler solution is already present, but possibly not obvious or poorly documented.
How much more obvious could it be? It’s in the Cloudwatch cron rules that the last parameter specifies the year.
We're taking this serverless thing too far. We're coming up with elaborate overcomplicated schemes to accomplish simple tasks just so we can say "we're not running a server for this". Instead of doing the simple thing and running a server process that periodically checks some queue for tasks to execute, this "scheme" involves triggering a lambda function every minute, to do some IO operations on a distributed file st…
Waking on items in queue would be better, though, and AWS has it built-in; the authors just did not use it for some reason.
Earlier quoted context omitted.
How much more obvious could it be? It’s in the Cloudwatch cron rules that the last parameter specifies the year.
Can you attach data to the event ?
https://docs.aws.amazon.com/cli/latest/reference/events/put-...
The data you attach will be included in the SNS message or the data that gets sent to the lambda handler.
You can put static JSON as an event from either the AWS Console, the CLI, or CloudFormation.
Earlier quoted context omitted.
Can you elaborate more? Is this related to S3 uptime SLA or is there a different reason?
I believe it is different to the S3 uptime SLA. You can measure the impact and potentially automate recovery of missed events by: 1) Keep a track of events published. 2) Generate an S3 inventory daily. 3) Compare events received to objects listed in inventory. You rarely end up with fewer events than objects, but it does occur. I've personally only observed this with SNS target, but due to it being a problem with S3,…
A much more resilient approach would be:
S3 event -> SNS Topic -> SQS Queue -> lambda.
and set up a dead letter queue for the SQS queue.
It doesn’t help with the reliability of S3 events (and I’ve never seen that happen), but it does help if their is an error running your lambda.
Move the S3 object after processing it. As long as you move it to a bucket in the same region, there aren’t any charges.
Then if you are really paranoid, you can have a timed lambda that checks the source S3 bucket periodically and manually sends SNS messages to the same topic to force processing.