see also aws athena https://aws.amazon.com/athena/ ?
Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
21–30 of 78 posts
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#22Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#23see also aws athena https://aws.amazon.com/athena/ ?
I did not know about this...looks like Amazon's version of BigQuery. Fantastic!
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#24Earlier quoted context omitted.
Ah yeah, just realizing this...what would you recommend?
I also came here thinking this is some sort of aws lambda triggered within the context of certain s3 file. I would say anyone who's heard of AWS lambda would think that way. Maybe functional-s3?
Okay this seems like a good alternative. I just renamed the repo. Renaming it on npm...is a bit cumbersome :|
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#25So... the client-side code iterates S3 objects matching a certain filter, and then schedules a lambda for each one of those objects. Is that right? Or does the iteration procedure itself is a lambda? Also, when you chain several operators together, where does the chaining happen? I'd like to understand where different parts of the code are being executed.
I thought this too. You could easily ship/eval the toString()'s of the functions in individual Lambda functions, the name is definitely confusing haha.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#26Getting aan index of (millions of) files on s3 is very slow for us, like, days. Is there anything you do to work around this? It seems since this is not an AWS Lambda project the client first has to acquire an index from S3 before concurrency benefits set in?
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#27Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#28Getting aan index of (millions of) files on s3 is very slow for us, like, days. Is there anything you do to work around this? It seems since this is not an AWS Lambda project the client first has to acquire an index from S3 before concurrency benefits set in?
This does not have to do with AWS Lambda, I'm thinking about renaming it to "functional-s3", or something similar.
To answer your question, there isn't really a workaround for this yet, although indexing should be much quicker than "days". All the keys are listed recursively before running the lambda expression locally. If you have a huge number of files, this can take several minutes, maybe hours depending on the scope.
A workaround I've been considering is using a generator function to list the keys; that way, the lambda expression can start immediately, generating keys as it needs them.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#29Is this susceptible to any of S3's eventual consistency constraints?
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#30Its weird how S3 seems to be the unwanted stepchild of AWS.
So many obvious innovations just aren't turning up.
For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind.
In fact it's strange that there is NO useful filtering at all apart from the very useful folder/hierarchy/prefix filtering. But apart from that you can't do wildcard searches or filters or date filters or tag filters.
I'm building an application right now that needs to get a list of all the jpg files - the only way to do that is get every single object in the bucket and manually filter out the unwanted ones - feels like its 1988 again.
It seems like it would also be valuable for there to be alternate interfaces to S3 such as the ability to send data via ftp or SMTP or sftp or whatever, but there are no such interfaces.
Hopefully Google will goad AWS into action on S3 innovation by implementing such features.