Earlier quoted context omitted.
You don't need the content length of what you are uploading for multipart uploads, see ( http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadIniti... ). For each part of a multipart upload that is sent, however, you do need to include the content-length header, so that may be what you are referring to. These have a minimum size set by amazon of 5 MB. With the https://github.com/rlmcpherson/s3gof3r the memory overh…
Yeah, having to buffer the stream in order to set the correct value for the content-length header on the last part of the multipart upload is what I was referring to. Your example using a 20MB part size is quite feasible. Thanks! Is it safe to use the master branch? Do you have build instructions? I haven't used go before.
Re: How to Save 90% on your S3 Bill
#101We use the version in the master branch at CodeGuard to transfer many terabytes into and out of S3 daily. While I hesitate to call it "production-ready", as that means different things to different people, we do use it in production with no issues. If you want to try it without having to install go, there are statically linked binaries for OSX and Linux Amd/64 linked on the github page that should run on any distro without having to install any dependencies. If you have any issues or questions, feel free to contact me at the email in my HN profile. I hope it works for your use case! :)