S3concurrent, yet another S3 syncing project

#1

Post by wingchen » Sat, Sep 19, 2015, 3:50 PM UTC

S3concurrent, yet another S3 syncing project
winstonchen.co

Re: S3concurrent, yet another S3 syncing project

#2

Post by hrez » Sun, Sep 20, 2015, 1:03 AM UTC

How does its performance compare to aws s3 sync? Note that you can adjust max_concurrent_requests which defaults to 10.

Re: S3concurrent, yet another S3 syncing project

#3

Post by grhmc » Sun, Sep 20, 2015, 1:10 AM UTC

hrez wrote: post #2

How does its performance compare to aws s3 sync? Note that you can adjust max_concurrent_requests which defaults to 10.

I'm always surprised by the number of "download from s3" tools there are. AWS-CLI does a really quite nice job of it.

Re: S3concurrent, yet another S3 syncing project

#4

Post by hrez » Sun, Sep 20, 2015, 1:52 AM UTC

grhmc wrote: post #3

hrez wrote: post #2

How does its performance compare to aws s3 sync? Note that you can adjust max_concurrent_requests which defaults to 10.

I'm always surprised by the number of "download from s3" tools there are. AWS-CLI does a really quite nice job of it.

To be fair, here is a tool that has features no other tool I know: https://github.com/rlmcpherson/s3gof3r It can saturate your bandwidth and you can pipe to/from it. It's great to deal with hundreds GB's archive on the fly. "Examples: $ tar -cf - /foo_dir/ | gof3r put -b my_s3_bucket -k bar_dir/s3_object $ gof3r get -b my_s3_bucket -k bar_dir/s3_object | tar -x

It's written in golang. So it's a simple binary.