OR You can move from S3 to some cheap servers with lot of data bandwidth.
How to Save 90% on your S3 Bill
41–50 of 103 posts
Re: How to Save 90% on your S3 Bill
#42interestingly enough, here's the original commit that defaulted `validate=True`: https://github.com/boto/boto/commit/95939debc3813468264159d5... EDIT : Looks like the original committer is an Amazon employee.
> interestingly enough, here's the original commit that defaulted `validate=True`:
It's a commit which specifically added validate to allow skipping validation. If you read the diff, the call originally unconditionally performed the validation call.
Re: How to Save 90% on your S3 Bill
#43I really hated working with boto for s3. The abstractions they chose are really ambiguous and it feels like it's fighting against the underlying api. If you're building anything on top of s3, it might be easier to write a thin S3 REST client and use it directly instead of going through Boto. So many fewer surprises, and no more digging around in the boto source trying to figure out what so and so function actually does.
Re: How to Save 90% on your S3 Bill
#44Good news for django-storages users – this is off by default: https://bitbucket.org/david/django-storages/src/cb7366693ce1...
As far as I can see, that means this code is broken: if no validation is done it won't raise S3ResponseError (because there's no S3 query involved), and will never raise `ImproperlyConfigured`. And if you ask for your buckets to be auto-created, it will silently do one more request per (existing) bucket, on top of the auto-creation one.
And the conditional inside the except is useless, of course, since you can only be in the except if `auto_create_bucket` is set.
Re: How to Save 90% on your S3 Bill
#45Also see MimicDB [0]. Runs a transparent cache that responds to most S3 API calls locally using Redis to store metadata. Besides the cost savings, it's extremely fast. Listing an entire bucket with millions of objects is close to instantaneous. [0] http://mimicdb.com/
It also adds another piece of infrastructure that needs to be maintained and can go down. Not necessarily the best option for everyone.
While the caching would be more effective with one large centralized instance, I think the intended use case is to have one cache per server. So then it's not really extra infrastructure.
Re: How to Save 90% on your S3 Bill
#46Re: How to Save 90% on your S3 Bill
#47Earlier quoted context omitted.
It also adds another piece of infrastructure that needs to be maintained and can go down. Not necessarily the best option for everyone.
While the caching would be more effective with one large centralized instance, I think the intended use case is to have one cache per server. So then it's not really extra infrastructure.
Actually, you can do it either way since it's backed by Redis. You can set all servers to connect to the same Redis instance, or run them all individually.
Re: How to Save 90% on your S3 Bill
#48Are you sure this counts as a LIST request? Technically the method of the list bucket API call is GET.
Re: How to Save 90% on your S3 Bill
#49Yikes! That's a horrible thing for a library to do to its users. This definitely should be changed in the library. I noticed that no such Issue exists, so I opened one. https://github.com/boto/boto/issues/2078
I kind of agree with garnaat's reply, if they just suddenly change the default from TRUE to FALSE they're going to break backwards compatibility with anyone using the library and worse still in a really subtle way! All they can really do at this stage is add a warning to the documentation and hope that new people using the library figure out the significance.
Or issue an actual runtime warning: http://docs.python.org/2/library/warnings.html
Re: How to Save 90% on your S3 Bill
#50Yikes! That's a horrible thing for a library to do to its users. This definitely should be changed in the library. I noticed that no such Issue exists, so I opened one. https://github.com/boto/boto/issues/2078
We have been using this in production for over 2 years at scale and just cross checked to have this enabled.
Totally agree, that this should be default False!