How to Save 90% on your S3 Bill • Hacker News Board

Re: How to Save 90% on your S3 Bill

#71

Post by Negitivefrags » Wed, Feb 05, 2014, 8:36 PM UTC

UnoriginalGuy wrote: post #29

jameskilton wrote: post #14

Yikes! That's a horrible thing for a library to do to its users. This definitely should be changed in the library. I noticed that no such Issue exists, so I opened one. https://github.com/boto/boto/issues/2078

I kind of agree with garnaat's reply, if they just suddenly change the default from TRUE to FALSE they're going to break backwards compatibility with anyone using the library and worse still in a really subtle way! All they can really do at this stage is add a warning to the documentation and hope that new people using the library figure out the significance.

That's why you should avoid using default arguments in library functions.

Re: How to Save 90% on your S3 Bill

#72

Post by cmer » Wed, Feb 05, 2014, 8:51 PM UTC

nathancahill wrote: post #31

cmer wrote: post #30

Earlier quoted context omitted.

It stores everything but the object

Yes. If you're storing the objects on your server, why would you use S3?

Caching. We request the same objects quite frequently and bandwidth is killing us.

Re: How to Save 90% on your S3 Bill

#73

Post by valtron » Wed, Feb 05, 2014, 9:06 PM UTC

You only need to worry about this if you're doing 100k+ LIST requests. For me, turning off validate would save 0.2%.

Re: How to Save 90% on your S3 Bill

#74

Post by jhenkens » Wed, Feb 05, 2014, 10:13 PM UTC

Negitivefrags wrote: post #71

UnoriginalGuy wrote: post #29

Earlier quoted context omitted.

I kind of agree with garnaat's reply, if they just suddenly change the default from TRUE to FALSE they're going to break backwards compatibility with anyone using the library and worse still in a really subtle way! All they can really do at this stage is add a warning to the documentation and hope that new people using the library figure out the significance.

That's why you should avoid using default arguments in library functions.

Great point! I have never heard that, but it makes perfect sense. Hopefully I will remember in the future.

Re: How to Save 90% on your S3 Bill

#75

Post by goldenkey » Wed, Feb 05, 2014, 10:21 PM UTC

saurik wrote: post #57

I found this exact same issue a few years ago after determining I had a reasonably-long-existing $75/day "leak" in my S3 expenses :/. (Seriously: $75/day, a number I am not exaggerating at all.) > 09:00:04 dls: I am performing millions of ListBucket request to this one amazon bucket every day > 09:00:25 d'oh > 09:00:28 adding up to almost 1.5 gigabytes of data traffic in/out on just those requests > 09:00:41 I have N…

Even the infamous saurik got floor-hit. closes jaw

Re: How to Save 90% on your S3 Bill

#76

Post by midas007 » Wed, Feb 05, 2014, 10:30 PM UTC

How to save 95%: don't use AWS long-term, buy dedicated hardware. http://blog.backblaze.com/2013/02/20/180tb-of-good-vibration...

Re: How to Save 90% on your S3 Bill

#77

Post by midas007 » Wed, Feb 05, 2014, 10:31 PM UTC

nathancahill wrote: post #11

notfunk wrote: post #3

interestingly enough, here's the original commit that defaulted `validate=True`: https://github.com/boto/boto/commit/95939debc3813468264159d5... EDIT : Looks like the original committer is an Amazon employee.

Before you jump on the conspiracy train, most Boto contributors are Amazon employees.

Yeah don't hate on em, Boto folks are pretty helpful.

Re: How to Save 90% on your S3 Bill

#78

Post by midas007 » Wed, Feb 05, 2014, 10:33 PM UTC

nathancahill wrote: post #62

watty wrote: post #56

Earlier quoted context omitted.

Why are you comparing S3 costs to "just" HDDs? HDD's aren't web enabled and require other hardware to run and keep running.

A more accurate comparison would be a BackBlaze Storage Pod [0] [0] http://blog.backblaze.com/2013/02/20/180tb-of-good-vibration...

CapEx + OpEx together averages 3x retail cost of hw for a medium sized web shop, labor and electricity being the biggest costs in each respectively.

Re: How to Save 90% on your S3 Bill

#79

Post by kennywinker » Wed, Feb 05, 2014, 10:42 PM UTC

gkumartvm wrote: post #41

OR You can move from S3 to some cheap servers with lot of data bandwidth.

There are a lot of use cases where s3 makes sense. For a small operation (one or two people) the admin time cost alone makes s3 pretty appealing. Also, the cost of a colo + server + redundant drives is a big initial spend when I have no way to know if my idea/app is going to get traction. Self-hosted may be cheaper for larger scales, but for a couple GBs s3 is going to win every time.

Re: How to Save 90% on your S3 Bill

#80

Post by rhizome » Wed, Feb 05, 2014, 11:24 PM UTC

cmircea wrote: post #54

nathancahill wrote: post #47

Earlier quoted context omitted.

Actually, you can do it either way since it's backed by Redis. You can set all servers to connect to the same Redis instance, or run them all individually.

What if Redis goes down?

There's always a single point of failure.