Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Save 90% on your S3 Bill

appneta.com

Post reply on HN
Print view View original

51–60 of 103 posts

Re: How to Save 90% on your S3 Bill

#51

Post by toomuchtodo »

jameskilton wrote: post #14

Yikes! That's a horrible thing for a library to do to its users. This definitely should be changed in the library. I noticed that no such Issue exists, so I opened one. https://github.com/boto/boto/issues/2078

James,

It looks like a HEAD request can fulfill the same purpose at a much lower cost (this is mentioned by a comment on your Issue: (comment by kislyuk)

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEA...

Shall I make the change and pull request it? I didn't want to duplicate effort.

Re: How to Save 90% on your S3 Bill

#52

Post by fours »

nathancahill wrote: post #47
bgwhn wrote: post #45

Earlier quoted context omitted.

While the caching would be more effective with one large centralized instance, I think the intended use case is to have one cache per server. So then it's not really extra infrastructure.

Actually, you can do it either way since it's backed by Redis. You can set all servers to connect to the same Redis instance, or run them all individually.

Do you use Redis clustering in this case (otherwise how would the cache stay consistent)?

Re: How to Save 90% on your S3 Bill

#53

Post by toomuchtodo »

jeffbarr wrote: post #19
notfunk wrote: post #13

Earlier quoted context omitted.

Agreed, I doubt Amazon had any evil intentions with this change. I wonder more if this was a product of Amazon developers using S3 (i.e. dogfooding) and not noticing the cost side effect because I'm assuming they don't get billed?

We actually do get billed, but we don't have to pay. I always check my bill to make sure that I am not using any resources that I don't need. I also pay for my own personal EC2 instance and about 350 GB of S3 storage. Begin a genuine user and customer of AWS helps me to be a better employee.

Jeff,

You may want to check LIST request statistics over the next few weeks. Between this thread, an Issue for boto, etc. I'm curious if you see a noticeable decline in LIST requests with the attention this has brought. I'm just curious from a data standpoint.

Re: How to Save 90% on your S3 Bill

#54

Post by cmircea »

nathancahill wrote: post #47
bgwhn wrote: post #45

Earlier quoted context omitted.

While the caching would be more effective with one large centralized instance, I think the intended use case is to have one cache per server. So then it's not really extra infrastructure.

Actually, you can do it either way since it's backed by Redis. You can set all servers to connect to the same Redis instance, or run them all individually.

What if Redis goes down?

Re: How to Save 90% on your S3 Bill

#57

Post by saurik »

I found this exact same issue a few years ago after determining I had a reasonably-long-existing $75/day "leak" in my S3 expenses :/. (Seriously: $75/day, a number I am not exaggerating at all.)

> 09:00:04 dls: I am performing millions of ListBucket request to this one amazon bucket every day

> 09:00:25 d'oh

> 09:00:28 adding up to almost 1.5 gigabytes of data traffic in/out on just those requests

> 09:00:41 I have NO CLUE what could POSSIBLY doing even a SINGLE ListBucket request on that bucket

> 09:00:49 LOL

Re: How to Save 90% on your S3 Bill

#59

Post by yelnatz »

notfunk wrote: post #3

interestingly enough, here's the original commit that defaulted `validate=True`: https://github.com/boto/boto/commit/95939debc3813468264159d5... EDIT : Looks like the original committer is an Amazon employee.

The commit added another parameter so it is possible to skip the expensive get_all_keys call.

It is defaulted to true so that old code that called get_bucket() won't break. (Since old code would have used get_bucket with only one parameter.)

It was always true before that commit.

Therefore, in essence, an Amazon employee made it possible to save 90% of your S3 bills.

Re: How to Save 90% on your S3 Bill

#60

Post by rosser »

[deleted] wrote: post #55

[deleted]

Yeah, and the op-ex of a HDD just sitting there is like, zero, man. Totes cheaper than S3.

(You aren't just paying for raw HDD capacity when you give Amazon money to use S3. You're also paying for many of the things between that capacity and you, as well.)

Post reply on HN