see also aws athena https://aws.amazon.com/athena/ ?
Somehow I'd overlooked this too: nice find
Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
41–50 of 78 posts
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#42see also aws athena https://aws.amazon.com/athena/ ?
Somehow I'd overlooked this too: nice find
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#43First impression: this is a brilliant piece of software design. The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow. The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 th…
... Except it's not! The "lambda" here isn't AWS Lambda. It's a locally executed function. Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then yes that'd be awesome.
Bah. My first impression was totally wrong in that case. Here's hoping someone builds a version of this that executes magically in the lambda cloud.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#44Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…
While great points, I think it might then go beyond the "Simple" in the S3 name itself. Wasn't the original purpose of the service to have it as a dumb storage, and you'll layer metadata as required? I.e. storing indices separately with whatever functionality is needed (be it date/path filtering).
Surely some extra functionality would not obfuscate the inherent simplicity of S3.
A an S3Query module would not, I think, make things harder for S3 users.
And frankly - it would be awesome.
I used s3 a lot, and loathe to switch to a DB if I can avoid it.
Some querying and indexing features I think would be taken up by a large number of devs.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#45see also aws athena https://aws.amazon.com/athena/ ?
That seems cool but paying per query (per TB scanned) frightens me. I imagine having to fret about how efficient my queries are...
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#46Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…
S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store.
I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" prefix, list requests are impossible to parallelize efficiently (because you can't subdivide the work), and our scripts took weeks to run to completion. Hard-earned experience: If you're storing large quantities of stuff in S3, always pick a shardable prefix. The upload date is a good choice. A random string will also do.
After this, my solution for any non-trivially-sized storage use case is to store an inventory of objects separately in a performant PostgreSQL database, and make sure all writes go through a service layer that shields the consumer from the details of S3. This has some benefits over a hypothetical centralized approach (but some downsides, like the possibility that things get out of sync if you sidestep the inventory). Overall, I wish S3 would store its metadata in something like BigQuery.
Anyone know if Google Cloud Platform's S3 equivalent, Cloud Storage, improves on these issues?
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#47Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…
S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…
I wonder if "bucket notifications" are reliable enough that one could keep such an index DB populated automatically?
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#48If anyone is interested in this same kind of architecture for multi-cloud file-system providers ( no cloud lock-in ), please check out this project: https://github.com/bigcompany/hook.io-vfs
Used in production, but it could use some contributors.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#49I thinking having the default be destructive for mapping is a strange design decision. That is going to bite someone one day soon.
Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
#50Earlier quoted context omitted.
S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…
I wonder if "bucket notifications" are reliable enough that one could keep such an index DB populated automatically?
I think SQS would be reliable enough here, yes.