Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3sync – Tool belt for managing your S3 buckets

github.com

Post reply on HN
Print view View original

1–10 of 22 posts

S3sync – Tool belt for managing your S3 buckets

#1

Post by clarete »

I started contributing to s3sync before hearing about the official amazon s3 tool written in python. However, after testing the official tool for a couple minutes, I decided to spend more time working on s3sync and here's the result.

Thanks to Michael Grosser for his support and patches!

S3sync – Tool belt for managing your S3 buckets
github.com

Re: S3sync – Tool belt for managing your S3 buckets

#2

Post by kcorbitt »

This is handy and extends S3's use case by making two-way sync straightforward. My long-term storage is in Amazon Glacier; I wonder how much effort it would take to extend this and make a straightforward process to pull the data from Glacier, sync with S3 and then push back to Glacier.

Re: S3sync – Tool belt for managing your S3 buckets

#4

Post by mfonda »

How does this compare to the existing s3cmd [1] tool? One issue I've had with s3cmd is it tends to use a lot of memory when syncing two large buckets. I'd love to see a tool that was faster / more memory efficient than s3cmd. Would be awesome to see some benchmarks and feature comparison in the README!

[1] http://s3tools.org/s3cmd

Re: S3sync – Tool belt for managing your S3 buckets

#5

Post by rsync »

mfonda wrote: post #4

How does this compare to the existing s3cmd [1] tool? One issue I've had with s3cmd is it tends to use a lot of memory when syncing two large buckets. I'd love to see a tool that was faster / more memory efficient than s3cmd. Would be awesome to see some benchmarks and feature comparison in the README! [1] http://s3tools.org/s3cmd

Agreed. We support s3cmd:

  ssh user@rsync.net s3cmd get s3://rsynctest/mscdex.exe
... and have for some time ... but we'd be happy to have some diversity there. One thing that sticks out immediately, of course, is that s3cmd is written in python and s3sync is a ruby gem. This makes things difficult since we insist on having no interpreters in our customer chroot...[1]

[1] We "freeze" things like s3cmd and rdiff-backup into binary executables so they don't require python in the environment

Re: S3sync – Tool belt for managing your S3 buckets

#6

Post by clarete »

mfonda wrote: post #4

How does this compare to the existing s3cmd [1] tool? One issue I've had with s3cmd is it tends to use a lot of memory when syncing two large buckets. I'd love to see a tool that was faster / more memory efficient than s3cmd. Would be awesome to see some benchmarks and feature comparison in the README! [1] http://s3tools.org/s3cmd

So, basically, the main difference is the language they're implemented in. I started working on s3sync cause I found its code easier to read and I really needed to practice my ruby skills.

Although s3cmd is older and has more features, s3sync was designed to grow stable and well tested. I'm definitely planning to write benchmarks to s3sync and improve its performance as much as I can.

Re: S3sync – Tool belt for managing your S3 buckets

#9

Post by eterps »

mfonda wrote: post #4

How does this compare to the existing s3cmd [1] tool? One issue I've had with s3cmd is it tends to use a lot of memory when syncing two large buckets. I'd love to see a tool that was faster / more memory efficient than s3cmd. Would be awesome to see some benchmarks and feature comparison in the README! [1] http://s3tools.org/s3cmd

The speed of https://github.com/cobbzilla/s3s3mirror is quite impressive, I suggest you give it a try.
Post reply on HN