S5cmd: Parallel S3 and local filesystem execution tool
1–10 of 28 posts
Re: S5cmd: Parallel S3 and local filesystem execution tool
#2I posted a link to s5cmd in the other post about s3mini. Really want to try it out migrating a few TBs of s3 data.
Re: S5cmd: Parallel S3 and local filesystem execution tool
#3Very interesting. And this is an amazing graph for small file uploading/download speed improvement. I have the feeling that all cloud drives are really not optimised for many small files like smaller than 1Mbytes in average.
https://raw.githubusercontent.com/peak/s5cmd/master/doc/benc...
I've implemented at work once a rudimentary parallel uploading of many small files to S3 in Python and with boto3 (was not allowed to use a third party library or tool at that time) because it's soooo slow to upload many small files to S3. It really takes ages and even if you just upload 8 small files in parallel it makes a huge difference.
Re: S5cmd: Parallel S3 and local filesystem execution tool
#4> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s)
I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3.
My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 file is capped at 1.6GB/s (at least per ec2 instance). At least I have never managed myself nor seen any tool capable of going beyond that.
My understanding is then that this benchmark's claims of 4.3GB/s are across multiple files, but then it would be rather meaningless, as it's free concurrency basically.
Re: S5cmd: Parallel S3 and local filesystem execution tool
#5> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…
[deleted]
Re: S5cmd: Parallel S3 and local filesystem execution tool
#6[flagged]
Re: S5cmd: Parallel S3 and local filesystem execution tool
#7If you don't want to install, or maintain, s5cmd yourself:
ssh user@rsync.net s5cmd
Password:
NAME:
s5cmd - Blazing fast S3 and local filesystem execution tool
USAGE:
s5cmd [global options] command [command options] [arguments...]
Re: S5cmd: Parallel S3 and local filesystem execution tool
#8> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…
I think that's the point of this tool, that pipelining and parallelism are necessary to get high performance out of S3.
Re: S5cmd: Parallel S3 and local filesystem execution tool
#9> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…
[deleted]
Re: S5cmd: Parallel S3 and local filesystem execution tool
#10> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…
I cant verify 40Gbps because I have never had access to a pipe that fast, but when I ran this tool on an AWS instance with a 20Gbps connection, it saturated that easily and maintained that speed for the duration of the transfer.