Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Save 90% on your S3 Bill

appneta.com

Post reply on HN
Print view View original

21–30 of 103 posts

Re: How to Save 90% on your S3 Bill

#21

Post by notfunk »

jeffbarr wrote: post #19
notfunk wrote: post #13

Earlier quoted context omitted.

Agreed, I doubt Amazon had any evil intentions with this change. I wonder more if this was a product of Amazon developers using S3 (i.e. dogfooding) and not noticing the cost side effect because I'm assuming they don't get billed?

We actually do get billed, but we don't have to pay. I always check my bill to make sure that I am not using any resources that I don't need. I also pay for my own personal EC2 instance and about 350 GB of S3 storage. Begin a genuine user and customer of AWS helps me to be a better employee.

Ahh cool! I completely agree that using your own products from a customer point of makes a better employee. Thanks for the input =)

Re: How to Save 90% on your S3 Bill

#22

Post by ducci »

This has been in the boto library for years, across multiple services. I noticed it when first using SimpleDB, switching it on in a production environment was much more expensive than we had originally calculated. I noticed the bizarre "Domain Validate" calls after pouring through logs of all boto activity.

The boto guys have justified it in the past: https://groups.google.com/forum/#!topic/boto-users/1DVfbo4CD...

I still don't agree with their reasoning to leave it default, once you try to do something on a non-existent domain/bucket it will throw an error anyway, and I would argue the "extra work" is much cheaper than leaving these defaults on which I would expect are completely redundant for most users.

Re: How to Save 90% on your S3 Bill

#24

Post by tszming »

engates wrote: post #8

This is one of the reasons why Rackspace simplified pricing of Cloud Files from the start. No fees for PUT, POST, LIST, HEAD, GET, DELETE...no extra fees for Akamai CDN requests. Very simple with no hidden fees that surprise you at the end of the month.

>> No fees for PUT, POST, LIST, HEAD, GET, DELETE...no extra fees for Akamai CDN requests

or you can say the fee is bundled, i.e.

(Rackspace) vs (S3 us-east-1)

Storage: 0.105/GB/mo vs 0.085/GB/mo

Bandwidth out: 0.20/GB/mo vs 0.120/GB/mo

Re: How to Save 90% on your S3 Bill

#25

Post by rmc »

engates wrote: post #8

This is one of the reasons why Rackspace simplified pricing of Cloud Files from the start. No fees for PUT, POST, LIST, HEAD, GET, DELETE...no extra fees for Akamai CDN requests. Very simple with no hidden fees that surprise you at the end of the month.

Those fees for "operations" are there for a good reason. Otherwise, us smart techies would hack it.

I heard a talk by someone at a mega tech company that has their own internal cloud for their teams, and they "charge" each team based on usage. One team stored lots of file with 12,000 character filenames with zero contents. Since the company only "charged" for file size, that team had a tiny charge!

Re: How to Save 90% on your S3 Bill

#26

Post by penguindev »

That's a crap default and a crap name. It should be prefetch_all_keys=False. (edit: and some documented reason WHY you would want to do such a thing)

I ran into this recently when making my own s3 sync tool, because the commonly used tool is completely broken (requires something called a 'config file' to function). But I didn't pay it too much mind, because I forgot the price discrepancy for ListBucket calls.

PS if you want to see what boto is doing do this: logging.basicConfig(filename="boto.log", level=logging.DEBUG)

Re: How to Save 90% on your S3 Bill

#27

Post by ducci »

notfunk wrote: post #3

interestingly enough, here's the original commit that defaulted `validate=True`: https://github.com/boto/boto/commit/95939debc3813468264159d5... EDIT : Looks like the original committer is an Amazon employee.

That's a change by the original author of boto. I don't think he was an Amazon employee that far back...

Re: How to Save 90% on your S3 Bill

#29

Post by UnoriginalGuy »

jameskilton wrote: post #14

Yikes! That's a horrible thing for a library to do to its users. This definitely should be changed in the library. I noticed that no such Issue exists, so I opened one. https://github.com/boto/boto/issues/2078

I kind of agree with garnaat's reply, if they just suddenly change the default from TRUE to FALSE they're going to break backwards compatibility with anyone using the library and worse still in a really subtle way!

All they can really do at this stage is add a warning to the documentation and hope that new people using the library figure out the significance.

Post reply on HN