Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S5cmd: Parallel S3 and local filesystem execution tool

github.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: S5cmd: Parallel S3 and local filesystem execution tool

#11

Post by bob1029 »

Galanwe wrote: post #4

> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…

I pushed a very large file to S3 the other day from my local machine and I was barely seeing 90 megabytes/s with 5gbps of upstream available.

I wonder what the limit is for uploads inside AWS. I've never been able to afford EC2 instances with the requisite networking or I/O performance.

Re: S5cmd: Parallel S3 and local filesystem execution tool

#12

Post by rdg42 »

rsync wrote: post #7

If you don't want to install, or maintain, s5cmd yourself: ssh user@rsync.net s5cmd Password: NAME: s5cmd - Blazing fast S3 and local filesystem execution tool USAGE: s5cmd [global options] command [command options] [arguments...] If you move data between clouds, you don't need to use any of your own bandwidth ...

s5cmd is not listed here https://www.rsync.net/resources/howto/remote_commands.html ? :-)

Re: S5cmd: Parallel S3 and local filesystem execution tool

#14

Post by beastman82 »

Galanwe wrote: post #4

> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…

It works with single files

Also you should up your chunk size to limit tps

Source: running s5cmd in prod for 1yr+

Re: S5cmd: Parallel S3 and local filesystem execution tool

#18

Post by vasco »

rdg42 wrote: post #12
rsync wrote: post #7

If you don't want to install, or maintain, s5cmd yourself: ssh user@rsync.net s5cmd Password: NAME: s5cmd - Blazing fast S3 and local filesystem execution tool USAGE: s5cmd [global options] command [command options] [arguments...] If you move data between clouds, you don't need to use any of your own bandwidth ...

s5cmd is not listed here https://www.rsync.net/resources/howto/remote_commands.html ? :-)

It either was added in the meantime or it already was there

> Backup Tools

> attic, borg (usage notes), rclone, unison, s5cmd and git are installed (on our server side).

> You would most likely call these commands by running these tools locally and connecting to rsync.net with them.

Re: S5cmd: Parallel S3 and local filesystem execution tool

#19

Post by hknlof1 »

Galanwe wrote: post #4

> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 fil…

It is definitely possible. Depends on how many 503s you might need to hit and when there heuristic model allows you to increase throughput. As a customer, you can also push for higher dedicated throughput.

Re: S5cmd: Parallel S3 and local filesystem execution tool

#20

Post by quodlibetor »

I recently wrote a similar tool focused more on optimizing the case of exploring millions or billions of objects when you know a few aspects of the path: https://github.com/quodlibetor/s3glob

It supports glob patterns like so, and will do smart filtering at every stage possible: */2025-0[45]-*/user*/*/object.txt

I haven't done real benchmarks, but it's parallel enough to hit s3 parallel request limits/file system open file limits when downloading.*

Post reply on HN