71–78 of 78 posts
Earlier quoted context omitted.
I wonder if "bucket notifications" are reliable enough that one could keep such an index DB populated automatically?
Yes, just hook those up to a lambda function and write to dynamodb or something
Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…
S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…
Earlier quoted context omitted.
Yes, but are you sure your database matches the underlying data store? The real problem with building a metadata index outside is that you then have the synchronization validation - yuk.
You can always do a full scan of your S3 namespace every week or so and synchronize the index. This gives your consumers low latency access to the object store, as index lookups are extremely fast, it minimizes the cost of lookup events on S3.
Earlier quoted context omitted.
S3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" pre…
Have you looked into the inventory functionality? It was just added last November. http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inven...
Sounds a bit like something they cooked up in a hurry to avoid having to design a BigQuery-type service for querying arbitrary metadata; I bet they had some huge customer with a need to get a CSV file for a bucket, that were willing to effectively bankroll the development of this feature.
But yes. That would sidestep the issue. You'd still have to turn on the feature and wait for the CSV file to build (apparently the best granularity is daily), of course, but it would help tremendously. Wish that had existed when we had our difficulties, about a year ago.
Earlier quoted context omitted.
You can always do a full scan of your S3 namespace every week or so and synchronize the index. This gives your consumers low latency access to the object store, as index lookups are extremely fast, it minimizes the cost of lookup events on S3.
So my database is up to a week wrong? Errr.....
the DB is only incomplete for as long it takes to commit to the SQL layer after storing successfully.
Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at a…
Earlier quoted context omitted.
That seems cool but paying per query (per TB scanned) frightens me. I imagine having to fret about how efficient my queries are...
It's not that bad. You can compress the data on S3 in ORC or Parquet format, and you only pay for the compressed data you read, so 1TB can be 130GB after compression. Plus, these formats store summary data, so queries like SELECT COUNT don't have to do a full table scan - they can read just a few KB of summary data for the result.
Earlier quoted context omitted.
It's not that bad. You can compress the data on S3 in ORC or Parquet format, and you only pay for the compressed data you read, so 1TB can be 130GB after compression. Plus, these formats store summary data, so queries like SELECT COUNT don't have to do a full table scan - they can read just a few KB of summary data for the result.
But that's a lot of work....Just to have sane costs for reads of your data
1. hive 2. INSERT INTO parquet_table SELECT * FROM csv_table;