If you want parallel upload/download over s3 and other protocols (GDrive, ftp, WebDAV, ...) you can use rclone.
For s3 mounts I would use geesefs.
Have to later take a look at s5cmd as well...
S5cmd: Parallel S3 and local filesystem execution tool
21–28 of 28 posts
Re: S5cmd: Parallel S3 and local filesystem execution tool
#22> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…
I cant verify 40Gbps because I have never had access to a pipe that fast, but when I ran this tool on an AWS instance with a 20Gbps connection, it saturated that easily and maintained that speed for the duration of the transfer.
I just spawned a r6a.16xlarge with a 25gbps NIC, created a 10GB file which I uploaded to an S3 bucket in the same region, through a local S3 VPC endpoint.
Downloading that 10GB file to /dev/shm with s5cmd took 24s, all while spawning 20 or so threads which were all idling for IO.
The same test using a Python tool (http://github.com/NewbiZ/s3pd) with the same amount of workers took 10s.
Cranking up the worker count of the latter library until there is no more speedup, I can reach 6s with 80 workers. That is, 10/6 = 1.6GB/s, which seems to confirm my previous comment.
What am I doing wrong ?
Re: S5cmd: Parallel S3 and local filesystem execution tool
#23I have used s5cmd in a professional setting and it works wonderfully. I have never attempted to test performance to confirm their claims, but as an out of the box client, it is (anecdotally) significantly faster than anything else I have tried.
My only headache was that I was invoking it from python, and it does not have bindings, so I had to write a custom wrapper to call out to it. I am not sure of the difficulty of adding native support for Python, but I assume its not worth the squeeze and just calling out to a subprocess will work for most user's needs.
Re: S5cmd: Parallel S3 and local filesystem execution tool
#24> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…
The S3 API allows requests to read a byte range of the file (sorry , object). So you could have multiple connections each reading a different byte range. Then the ranges would need to be written to the target local file using a random access pattern.
Re: S5cmd: Parallel S3 and local filesystem execution tool
#25> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…
The S3 API allows requests to read a byte range of the file (sorry , object). So you could have multiple connections each reading a different byte range. Then the ranges would need to be written to the target local file using a random access pattern.
I know that already... and it is exactly what I tested and confirmed here https://news.ycombinator.com/item?id=44249137
You can spawn multiple connections to S3 to retrieve chunks of a file in parallel, but each of these connections is capped at 80MB/s, and the whole of these connections, while operating on a single file, to a single EC2 instance, is capped at 1.6GB/s.
Re: S5cmd: Parallel S3 and local filesystem execution tool
#26What is meant by "filesystem execution" here? Is it just an S3 file transfer tool?
Re: S5cmd: Parallel S3 and local filesystem execution tool
#27Very interesting. And this is an amazing graph for small file uploading/download speed improvement. I have the feeling that all cloud drives are really not optimised for many small files like smaller than 1Mbytes in average. https://raw.githubusercontent.com/peak/s5cmd/master/doc/benc... I've implemented at work once a rudimentary parallel uploading of many small files to S3 in Python and with boto3 (was not allowed…
Most S3 systems are terrible with small files (~50KB). 1MB can still be ok though
Re: S5cmd: Parallel S3 and local filesystem execution tool
#28Earlier quoted context omitted.
I cant verify 40Gbps because I have never had access to a pipe that fast, but when I ran this tool on an AWS instance with a 20Gbps connection, it saturated that easily and maintained that speed for the duration of the transfer.
I just spawned a r6a.16xlarge with a 25gbps NIC, created a 10GB file which I uploaded to an S3 bucket in the same region, through a local S3 VPC endpoint. Downloading that 10GB file to /dev/shm with s5cmd took 24s, all while spawning 20 or so threads which were all idling for IO. The same test using a Python tool ( http://github.com/NewbiZ/s3pd ) with the same amount of workers took 10s. Cranking up the worker count…
Okay I found the trick, buried in the benchmark setup of s5cmd.
The claimed numbers are _not_ reached with S3, but rather from a custom server emulating the S3 API, hosted on the client machine.
I think this is very misleading, since these benchmark numbers are not reachable in any real life scenario. It also shows that there is very little point in using s5cmd compared to other tools, since beyond 1.6GB/s the throttling will be from S3, not from the client, so any tool able to saturate 1.6GB/s will be enough.