Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

github.com

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#61

Post by philliphaydon »

lobster_johnson wrote: post #46
hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…

I did something similar storing the information in PostgreSQL but made the inserts/updates/deletes based on the events of s3. If an object was stored it would insert into the database. If it was deleted it would soft delete in the database. Worked out well for me.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#62

Post by philliphaydon »

hoodoof wrote: post #57
illumin8 wrote: post #53

Earlier quoted context omitted.

First, I wanted to say, you bring up some very good points. S3 wasn't really designed to be a searchable key/value store, as you have to pay for lookups, and pagination kills your ability to effectively search anything greater than a few thousand objects in a hierarchy, within a reasonable amount of time. There are, however, ways to solve this: you could fire a Lambda function whenever an object is put into your S3 b…

As mentioned elsewhere in this thread, an external metadata database of S3 object immediately introduces synching and validity issues. DOS is smarter than S3.

I believe it works fine.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#63

Post by afandian »

lobster_johnson wrote: post #46
hoodoof wrote: post #30

Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…

S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…

As someone heading down a similar path (and I'm fairly sure I've got sensible prefixes) can you share an example of a prefix that caused you trouble. Is it something like

    /path/to/big-dir/«lots-of-sequential-filenames»

?

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#64

Post by primax »

wellsjohnston wrote: post #60
cle wrote: post #29

Is this susceptible to any of S3's eventual consistency constraints?

I'm not aware of S3's consistency constraints. What are those?

http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction....

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#65

Post by bpicolo »

cle wrote: post #29

Is this susceptible to any of S3's eventual consistency constraints?

Anything you read from S3 is, so yes.

The best way to prevent eventual consistency issues in s3 is to use immutable files. Then you have consistency-now.

They don't have explicit SLAs on this, unfortunately, but I've heard rumored that internal pagers start firing with consistency behind on the order of hours.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#66

Post by akx »

afandian wrote: post #63
lobster_johnson wrote: post #46

Earlier quoted context omitted.

S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…

As someone heading down a similar path (and I'm fairly sure I've got sensible prefixes) can you share an example of a prefix that caused you trouble. Is it something like /path/to/big-dir/«lots-of-sequential-filenames» ?

I suppose it's like with regular file systems -- don't have too many files in a directory.

In your use case, consider `/path/to/big-dir/AA/AABB/AABBCC` or similar?

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#67

Post by afandian »

akx wrote: post #66
afandian wrote: post #63

Earlier quoted context omitted.

As someone heading down a similar path (and I'm fairly sure I've got sensible prefixes) can you share an example of a prefix that caused you trouble. Is it something like /path/to/big-dir/«lots-of-sequential-filenames» ?

I suppose it's like with regular file systems -- don't have too many files in a directory. In your use case, consider `/path/to/big-dir/AA/AABB/AABBCC` or similar?

Sorry, that example wasn't my data, it was to illustrate question.

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#68

Post by lobster_johnson »

afandian wrote: post #63
lobster_johnson wrote: post #46

Earlier quoted context omitted.

S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…

As someone heading down a similar path (and I'm fairly sure I've got sensible prefixes) can you share an example of a prefix that caused you trouble. Is it something like /path/to/big-dir/«lots-of-sequential-filenames» ?

Exactly. It works fine for most tasks, of course, but if you ever want to process the contents of the S3 bucket in bulk, nothing will ever be able to parallelize that one list request to /path/to/big-dir.

If you don't use the evenly-distributed-prefix trick, your only chance of speeding it up is knowing the file names beforehand. If they're all sequentially numbered, you might do that, of course.

The shardable prefix doesn't need to be at the top level. So you could also organize it like so, for example:

    /secret/documents/2016-01-01/00000001.doc

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#69

Post by noobiemcfoob »

hoodoof wrote: post #57
illumin8 wrote: post #53

Earlier quoted context omitted.

First, I wanted to say, you bring up some very good points. S3 wasn't really designed to be a searchable key/value store, as you have to pay for lookups, and pagination kills your ability to effectively search anything greater than a few thousand objects in a hierarchy, within a reasonable amount of time. There are, however, ways to solve this: you could fire a Lambda function whenever an object is put into your S3 b…

As mentioned elsewhere in this thread, an external metadata database of S3 object immediately introduces synching and validity issues. DOS is smarter than S3.

DOS is 32 bit...

Re: Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

#70

Post by afandian »

lobster_johnson wrote: post #68
afandian wrote: post #63

Earlier quoted context omitted.

As someone heading down a similar path (and I'm fairly sure I've got sensible prefixes) can you share an example of a prefix that caused you trouble. Is it something like /path/to/big-dir/«lots-of-sequential-filenames» ?

Exactly. It works fine for most tasks, of course, but if you ever want to process the contents of the S3 bucket in bulk, nothing will ever be able to parallelize that one list request to /path/to/big-dir. If you don't use the evenly-distributed-prefix trick, your only chance of speeding it up is knowing the file names beforehand. If they're all sequentially numbered, you might do that, of course. The shardable prefix…

Thanks! I've read the docs and blog posts, but it was interesting to see a real live antipattern.
Post reply on HN