Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

github.com

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#51

Post by lobster_johnson »

lobster_johnson wrote: post #46
hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…

Replying to myself: Disappointingly, it seems GCP's Cloud Storage is pretty much a carbon clone of S3 as far as the API is concerned, down to the prefix/delimiter-based search.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#52

Post by nolite »

jessaustin wrote: post #47
lobster_johnson wrote: post #46

Earlier quoted context omitted.

S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…

I wonder if "bucket notifications" are reliable enough that one could keep such an index DB populated automatically?

Yes, just hook those up to a lambda function and write to dynamodb or something

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#53

Post by illumin8 »

hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

First, I wanted to say, you bring up some very good points. S3 wasn't really designed to be a searchable key/value store, as you have to pay for lookups, and pagination kills your ability to effectively search anything greater than a few thousand objects in a hierarchy, within a reasonable amount of time.

There are, however, ways to solve this: you could fire a Lambda function whenever an object is put into your S3 bucket that simply adds a single row to a DynamoDB table with the object name, along with any additional metadata you might like to capture to assure data provenance. Then, to search, you can simply query the DynamoDB table.

As always, there are many basic building blocks at AWS, but you have to connect them together (like legos) before they become useful for most applications.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#54

Post by illumin8 »

hoodoof wrote: post #39
xtracto wrote: post #38

Earlier quoted context omitted.

This, we love S3. What we did is add a SQL tier for some of the data we are storing there in case we want to do some more structured operations.

Yes, but are you sure your database matches the underlying data store? The real problem with building a metadata index outside is that you then have the synchronization validation - yuk.

You can always do a full scan of your S3 namespace every week or so and synchronize the index. This gives your consumers low latency access to the object store, as index lookups are extremely fast, it minimizes the cost of lookup events on S3.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#55

Post by illumin8 »

simonw wrote: post #43
koolba wrote: post #11

Earlier quoted context omitted.

... Except it's not! The "lambda" here isn't AWS Lambda. It's a locally executed function. Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then yes that'd be awesome.

Bah. My first impression was totally wrong in that case. Here's hoping someone builds a version of this that executes magically in the lambda cloud.

Well, you could run it on a large EC2 instance (x1.32xlarge?!:O) and it would be running the lambdas on the cloud, technically... ;-)

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#56

Post by illumin8 »

dajohnson89 wrote: post #45
hayd wrote: post #21

see also aws athena https://aws.amazon.com/athena/ ?

That seems cool but paying per query (per TB scanned) frightens me. I imagine having to fret about how efficient my queries are...

It's not that bad. You can compress the data on S3 in ORC or Parquet format, and you only pay for the compressed data you read, so 1TB can be 130GB after compression. Plus, these formats store summary data, so queries like SELECT COUNT don't have to do a full table scan - they can read just a few KB of summary data for the result.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#57

Post by hoodoof »

illumin8 wrote: post #53
hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

First, I wanted to say, you bring up some very good points. S3 wasn't really designed to be a searchable key/value store, as you have to pay for lookups, and pagination kills your ability to effectively search anything greater than a few thousand objects in a hierarchy, within a reasonable amount of time. There are, however, ways to solve this: you could fire a Lambda function whenever an object is put into your S3 b…

As mentioned elsewhere in this thread, an external metadata database of S3 object immediately introduces synching and validity issues.

DOS is smarter than S3.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#58

Post by hoodoof »

illumin8 wrote: post #54
hoodoof wrote: post #39

Earlier quoted context omitted.

Yes, but are you sure your database matches the underlying data store? The real problem with building a metadata index outside is that you then have the synchronization validation - yuk.

You can always do a full scan of your S3 namespace every week or so and synchronize the index. This gives your consumers low latency access to the object store, as index lookups are extremely fast, it minimizes the cost of lookup events on S3.

So my database is up to a week wrong? Errr.....
Post reply on HN