Is there something similar to this that exists that also caches s3 objects locally?
Nginx can be used as a caching proxy:
How to Save 90% on your S3 Bill
91–100 of 103 posts
Re: How to Save 90% on your S3 Bill
#92Re: How to Save 90% on your S3 Bill
#93Earlier quoted context omitted.
Great point! I have never heard that, but it makes perfect sense. Hopefully I will remember in the future.
I don't agree, you should just know what they are. Defaults are helpful.
Until they change, and your code breaks, and you have to go figure out why. The point of not (silently) using defaults is that as long as you're explicit with your arguments, you won't get surprised by changing defaults. Of course, you might get surprised by any number of other changes, but you can at least reduce your failure surface area a little bit :-)
Re: How to Save 90% on your S3 Bill
#94Because of the lack of fast and reliable s3 clients out there, I built https://github.com/rlmcpherson/s3gof3r, that can do over 1 Gbps with ease, both multipart uploads and parallelized downloads. The killer feature, though, is streaming that enables things like gof3r get -b -k | tar -x to extract directories tarred directories or any other streaming application. It also provides end-to-end md5 integrity checking. For objects over a few GB, I haven't found anything matching in speed or reliability.
Re: How to Save 90% on your S3 Bill
#95Because of the lack of fast and reliable s3 clients out there, I built https://github.com/rlmcpherson/s3gof3r , that can do over 1 Gbps with ease, both multipart uploads and parallelized downloads. The killer feature, though, is streaming that enables things like gof3r get -b -k | tar -x to extract directories tarred directories or any other streaming application. It also provides end-to-end md5 integrity checking. F…
From when I was looking into doing something similar, I recall s3 needing to know the content-length of upload parts up front. How do you handle that for streaming? Do you buffer in memory up to the max part size so that you can give the correct content-length header for the last part? I ask because my uses would include low-memory VMs so I'm curious about the memory overhead.
Re: How to Save 90% on your S3 Bill
#96Earlier quoted context omitted.
> It should be prefetch_all_keys=False. It does not prefetch any key (maxkeys is set to 0), it performs a query on the bucket to validate that the bucket exists and blow up if the bucket does not exist. With validate=False, you can call get_bucket and get a bucket object where no remote bucket exists.
It sounds like an annoying limitation of the API that (apparently?) you can't cheaply validate whether a bucket exists. Two manual work-arounds that come to mind: - store a list of created buckets as keys in another bucket. - store a dummy file in each bucket you create. Either method allows you to check the existence of the bucket with a GET request rather than a more expensive LIST request, but both are hackish. It…
> It sounds like an annoying limitation of the API that (apparently?) you can't cheaply validate whether a bucket exists.
It looks like there now is: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEA...
I'm guessing (hoping?) that didn't exist back when the feature was added to Boto, 7 years ago: https://github.com/boto/boto/commit/8410c365ee0120e073bf00bd...
Re: How to Save 90% on your S3 Bill
#97Because of the lack of fast and reliable s3 clients out there, I built https://github.com/rlmcpherson/s3gof3r , that can do over 1 Gbps with ease, both multipart uploads and parallelized downloads. The killer feature, though, is streaming that enables things like gof3r get -b -k | tar -x to extract directories tarred directories or any other streaming application. It also provides end-to-end md5 integrity checking. F…
From when I was looking into doing something similar, I recall s3 needing to know the content-length of upload parts up front. How do you handle that for streaming? Do you buffer in memory up to the max part size so that you can give the correct content-length header for the last part? I ask because my uses would include low-memory VMs so I'm curious about the memory overhead.
You don't need the content length of what you are uploading for multipart uploads, see (http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadIniti...). For each part of a multipart upload that is sent, however, you do need to include the content-length header, so that may be what you are referring to. These have a minimum size set by amazon of 5 MB. With the https://github.com/rlmcpherson/s3gof3r the memory overhead of a streaming upload is approximately part size * concurrent uploads (plus a couple of buffers in the pool). So, for example, if you configure the part size to 20 MB and set concurrent uploads to 10, that would be about 220 MB of memory usage.
Re: How to Save 90% on your S3 Bill
#98Earlier quoted context omitted.
Don't you have too much to do?
I took the nickname because I always take on more than I have time for, yet find the time to complete everything that I take on.
No offense intended, just enjoyed the relationship between your nick and your jumping on a new task. I don't actually mean to discourage!
Re: How to Save 90% on your S3 Bill
#99I wonder how much savings we're talking about. I always worry when engineers start talking about saving money.
Depends on your scale. Another commenter on this thread was in the +$75/day range from this issue.
I don't worry about cultivating a connection between engineering and business needs.
Re: How to Save 90% on your S3 Bill
#100Earlier quoted context omitted.
From when I was looking into doing something similar, I recall s3 needing to know the content-length of upload parts up front. How do you handle that for streaming? Do you buffer in memory up to the max part size so that you can give the correct content-length header for the last part? I ask because my uses would include low-memory VMs so I'm curious about the memory overhead.
You don't need the content length of what you are uploading for multipart uploads, see ( http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadIniti... ). For each part of a multipart upload that is sent, however, you do need to include the content-length header, so that may be what you are referring to. These have a minimum size set by amazon of 5 MB. With the https://github.com/rlmcpherson/s3gof3r the memory overh…
Yeah, having to buffer the stream in order to set the correct value for the content-length header on the last part of the multipart upload is what I was referring to. Your example using a 20MB part size is quite feasible. Thanks! Is it safe to use the master branch? Do you have build instructions? I haven't used go before.