Earlier quoted context omitted.
Check out MimicDB [0] [0] http://mimicdb.com/
It stores everything but the object
Yes. If you're storing the objects on your server, why would you use S3?
How to Save 90% on your S3 Bill
31–40 of 103 posts
Re: How to Save 90% on your S3 Bill
#32If you read the source code it calls a method called get_all_keys. Please realize, this does NOT get all the keys in the bucket. Passed to it is the maxkeys=0 argument which means no keys are returned and a single list call is made.
Yes, it is still a waste of money, but just make sure you understand that it's not actually listing your entire bucket.
Re: How to Save 90% on your S3 Bill
#33[deleted]
Re: How to Save 90% on your S3 Bill
#34I discovered the 'validate' argument just recently.
My concern was not costs, it was latency and response times.
My server is generating a HTML page with signed URLs to resources on S3. The 'get_bucket' call adds a bit of latency as it contacts S3 and I was thinking, does it really need anything from S3 to generate signed URLs if I already know the exact key names, and am pretty sure the bucket exists? Well, it does not, and adding "validate=False" speeded up things noticeably.
Re: How to Save 90% on your S3 Bill
#35Yikes! That's a horrible thing for a library to do to its users. This definitely should be changed in the library. I noticed that no such Issue exists, so I opened one. https://github.com/boto/boto/issues/2078
[deleted]
Re: How to Save 90% on your S3 Bill
#36Earlier quoted context omitted.
It stores everything but the object
Yes. If you're storing the objects on your server, why would you use S3?
Well, if the consumer is the server, rather than the server then serving the S3 data to a client, then I could see the benefit in the server caching retrievals locally.
Re: How to Save 90% on your S3 Bill
#37Earlier quoted context omitted.
Before you jump on the conspiracy train, most Boto contributors are Amazon employees.
Agreed, I doubt Amazon had any evil intentions with this change. I wonder more if this was a product of Amazon developers using S3 (i.e. dogfooding) and not noticing the cost side effect because I'm assuming they don't get billed?
Mitch just started working for AWS ~1.5 years ago or so I think? That code was written long before he worked for Amazon.
Re: How to Save 90% on your S3 Bill
#38Yikes! That's a horrible thing for a library to do to its users. This definitely should be changed in the library. I noticed that no such Issue exists, so I opened one. https://github.com/boto/boto/issues/2078
I kind of agree with garnaat's reply, if they just suddenly change the default from TRUE to FALSE they're going to break backwards compatibility with anyone using the library and worse still in a really subtle way! All they can really do at this stage is add a warning to the documentation and hope that new people using the library figure out the significance.
This is what major version numbers are for.
Re: How to Save 90% on your S3 Bill
#39Also see MimicDB [0]. Runs a transparent cache that responds to most S3 API calls locally using Redis to store metadata. Besides the cost savings, it's extremely fast. Listing an entire bucket with millions of objects is close to instantaneous. [0] http://mimicdb.com/
It also adds another piece of infrastructure that needs to be maintained and can go down. Not necessarily the best option for everyone.
Re: How to Save 90% on your S3 Bill
#40That's a crap default and a crap name. It should be prefetch_all_keys=False. (edit: and some documented reason WHY you would want to do such a thing) I ran into this recently when making my own s3 sync tool, because the commonly used tool is completely broken (requires something called a 'config file' to function). But I didn't pay it too much mind, because I forgot the price discrepancy for ListBucket calls. PS if y…
> It should be prefetch_all_keys=False.
It does not prefetch any key (maxkeys is set to 0), it performs a query on the bucket to validate that the bucket exists and blow up if the bucket does not exist. With validate=False, you can call get_bucket and get a bucket object where no remote bucket exists.