Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

github.com

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#11

Post by koolba »

simonw wrote: post #2

First impression: this is a brilliant piece of software design. The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow. The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 th…

... Except it's not!

The "lambda" here isn't AWS Lambda. It's a locally executed function.

Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then yes that'd be awesome.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#12

Post by andkon »

wellsjohnston wrote: post #10
dschnurr wrote: post #5

Might make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!

Ah yeah, just realizing this...what would you recommend?

func-s3?

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#13

Post by glogla »

koolba wrote: post #11
simonw wrote: post #2

First impression: this is a brilliant piece of software design. The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow. The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 th…

... Except it's not! The "lambda" here isn't AWS Lambda. It's a locally executed function. Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then yes that'd be awesome.

That would be glorious.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#14

Post by avip »

This is a nice project. For real-world use cases, we have good alternatives:

1. Migrate s3 ==> gc and use BigQuery which does support udf

2. Register to databricks (I'm not affiliated)

3. (for the brave) poke aws support to implement udf on Athena

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#16

Post by wellsjohnston »

stolendog wrote: post #15

where actually you can use it ? in which cases? can you provide examples?

Sure. We have application logs that come in to s3 and are stored by date prefix. I have CRON jobs that run node scripts that do various counts/statistics.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#18

Post by cocktailpeanuts »

wellsjohnston wrote: post #10
dschnurr wrote: post #5

Might make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!

Ah yeah, just realizing this...what would you recommend?

I also came here thinking this is some sort of aws lambda triggered within the context of certain s3 file. I would say anyone who's heard of AWS lambda would think that way.

Maybe functional-s3?

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#19

Post by kmfrk »

dschnurr wrote: post #5

Might make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!

S3 also has a built-in function to perform a Lambda operation on each upload, which makes it more confusing.

I mean, the README is excellently written and makes clear what the project does, so it's not a big deal beyond the ambiguous name.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#20

Post by stevewilhelm »

cocktailpeanuts wrote: post #18
wellsjohnston wrote: post #10

Earlier quoted context omitted.

Ah yeah, just realizing this...what would you recommend?

I also came here thinking this is some sort of aws lambda triggered within the context of certain s3 file. I would say anyone who's heard of AWS lambda would think that way. Maybe functional-s3?

fp-4-s3?
Post reply on HN