Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
1–10 of 78 posts
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#2First impression: this is a brilliant piece of software design.
The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow.
The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 this looks like my ideal abstraction.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#3First impression: this is a brilliant piece of software design. The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow. The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 th…
Thank you :)
Writing this was a necessity for me, being a 1-person data team coming from a Node.js background.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#4First impression: this is a brilliant piece of software design. The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow. The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 th…
Yes, I concur! I am definitely trying this out. I have a couple use cases where I think lambda functions would be useful but I don't currently have the time to figure out how to write and execute them.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#5Might make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#6So... the client-side code iterates S3 objects matching a certain filter, and then schedules a lambda for each one of those objects. Is that right? Or does the iteration procedure itself is a lambda? Also, when you chain several operators together, where does the chaining happen?
I'd like to understand where different parts of the code are being executed.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#7Might make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!
[deleted]
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#8So... the client-side code iterates S3 objects matching a certain filter, and then schedules a lambda for each one of those objects. Is that right? Or does the iteration procedure itself is a lambda? Also, when you chain several operators together, where does the chaining happen? I'd like to understand where different parts of the code are being executed.
First, a list of keys is generated based on the set context (and modifier functions). "context" returns a Request object, allowing you to call a lambda function (each, forEach, map, reduce, filter). Each lambda function returns a Promise, allowing you to chain them together. They will operate over the same context, in sequence.
Edit: This is not related to aws lambda...sorry for the confusion
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#9So... the client-side code iterates S3 objects matching a certain filter, and then schedules a lambda for each one of those objects. Is that right? Or does the iteration procedure itself is a lambda? Also, when you chain several operators together, where does the chaining happen? I'd like to understand where different parts of the code are being executed.
On a quick page-down through the code, I think this is not related to AWS-Lambda, it's more 'local lambda' where the map etc is run locally.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#10Might make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!
Ah yeah, just realizing this...what would you recommend?