Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

github.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#31

Post by kot-behemoth »

koolba wrote: post #11
simonw wrote: post #2

First impression: this is a brilliant piece of software design. The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow. The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 th…

... Except it's not! The "lambda" here isn't AWS Lambda. It's a locally executed function. Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then yes that'd be awesome.

It should be fairly easily doable with Gordon (https://github.com/jorgebastida/gordon), and scheduling via CloudWatch Events. Or Airflow.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#32

Post by kot-behemoth »

hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

While great points, I think it might then go beyond the "Simple" in the S3 name itself. Wasn't the original purpose of the service to have it as a dumb storage, and you'll layer metadata as required? I.e. storing indices separately with whatever functionality is needed (be it date/path filtering).

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#33

Post by wellsjohnston »

hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

I've had the same frustration with S3, and the reasons you went over are what drove me to create this. It seems like they made S3 (10 years ago now?) and just forgot about it. There is no way to sort/filter or even list all the objects in a bucket without writing a recursive algorithm using one of their SDKs.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#34

Post by hoodoof »

kot-behemoth wrote: post #32
hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

While great points, I think it might then go beyond the "Simple" in the S3 name itself. Wasn't the original purpose of the service to have it as a dumb storage, and you'll layer metadata as required? I.e. storing indices separately with whatever functionality is needed (be it date/path filtering).

Perhaps true.

I'll never do that though because I'd have to use DynamoDB, which is a technology that is high on my list of "technologies that I am least enthused about".

Also, I really shouldn't have to go to all the work of creating and maintaining a metadata database and implementing a query API just because I want to do searches more powerful than "list all objects" - that's Amazon's job.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#35

Post by maxaf »

hoodoof wrote: post #34
kot-behemoth wrote: post #32

Earlier quoted context omitted.

While great points, I think it might then go beyond the "Simple" in the S3 name itself. Wasn't the original purpose of the service to have it as a dumb storage, and you'll layer metadata as required? I.e. storing indices separately with whatever functionality is needed (be it date/path filtering).

Perhaps true. I'll never do that though because I'd have to use DynamoDB, which is a technology that is high on my list of "technologies that I am least enthused about". Also, I really shouldn't have to go to all the work of creating and maintaining a metadata database and implementing a query API just because I want to do searches more powerful than "list all objects" - that's Amazon's job.

Moreover, even if you had gone to the trouble of building such an API, S3 still doesn't offer bulk operations, so you'd have to operate on each matching object... one object at a time.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#36

Post by hoodoof »

maxaf wrote: post #35
hoodoof wrote: post #34

Earlier quoted context omitted.

Perhaps true. I'll never do that though because I'd have to use DynamoDB, which is a technology that is high on my list of "technologies that I am least enthused about". Also, I really shouldn't have to go to all the work of creating and maintaining a metadata database and implementing a query API just because I want to do searches more powerful than "list all objects" - that's Amazon's job.

Moreover, even if you had gone to the trouble of building such an API, S3 still doesn't offer bulk operations, so you'd have to operate on each matching object... one object at a time.

This isn't such an issue because you can update the DynamoDB index using an AWS Lambda function on every putobject or removeobject event.

It's still not something I want to do, mainly because I'd have to touch DynamoDB but secondly because, well, why the heck doesn't AWS do it?

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#37

Post by lancefisher »

wellsjohnston wrote: post #10
dschnurr wrote: post #5

Might make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!

Ah yeah, just realizing this...what would you recommend?

Smap for S3 map. :)

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#38

Post by xtracto »

kot-behemoth wrote: post #32
hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

While great points, I think it might then go beyond the "Simple" in the S3 name itself. Wasn't the original purpose of the service to have it as a dumb storage, and you'll layer metadata as required? I.e. storing indices separately with whatever functionality is needed (be it date/path filtering).

This, we love S3. What we did is add a SQL tier for some of the data we are storing there in case we want to do some more structured operations.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#39

Post by hoodoof »

xtracto wrote: post #38
kot-behemoth wrote: post #32

Earlier quoted context omitted.

While great points, I think it might then go beyond the "Simple" in the S3 name itself. Wasn't the original purpose of the service to have it as a dumb storage, and you'll layer metadata as required? I.e. storing indices separately with whatever functionality is needed (be it date/path filtering).

This, we love S3. What we did is add a SQL tier for some of the data we are storing there in case we want to do some more structured operations.

Yes, but are you sure your database matches the underlying data store?

The real problem with building a metadata index outside is that you then have the synchronization validation - yuk.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#40

Post by LightskinKanye »

hoodoof wrote: post #36
maxaf wrote: post #35

Earlier quoted context omitted.

Moreover, even if you had gone to the trouble of building such an API, S3 still doesn't offer bulk operations, so you'd have to operate on each matching object... one object at a time.

This isn't such an issue because you can update the DynamoDB index using an AWS Lambda function on every putobject or removeobject event. It's still not something I want to do, mainly because I'd have to touch DynamoDB but secondly because, well, why the heck doesn't AWS do it?

What do you have against DynamoDB?
Post reply on HN