continuous sync - mutagen.io (maybe you could extract some of the libraries)
depends on if it's large or small files.
Ask HN: Best modern file transfer/synchronization protocol?
21–30 of 59 posts
Re: Ask HN: Best modern file transfer/synchronization protocol?
#22I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well
rclone is garbage if you need performance or just copy between two points you control.
it shines when you need to sprinkle your data over many "clouds"
Re: Ask HN: Best modern file transfer/synchronization protocol?
#23Use HTTP. For fault tolerance, use resumeable downloads or resumeable uploads. There is work at the IETF on resumeable uploads right now: https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumabl...
I've had terrible experiences downloading large files over HTTP. I'm not sure why, range requests don't seem to be reliable or well supported. Something like BitTorrent is much better for large files: it divides the large file into chunks and hashes each chunk, and by default the chunks are downloaded in random order. BitTorrent seems much more reliable than range requests.
Same here. I downloaded some ~100Gb files, and everytime the connection broke, after relaunching, the shasum never matched. Even when the file size was exactly the same that the server reported.
Re: Ask HN: Best modern file transfer/synchronization protocol?
#24If it’s one way (that wasn’t quite clear from the requirements to me). take a look at https://tus.io/
This is awesome! I am indeed looking for something one way - so this looks great. Thanks for sharing.
Re: Ask HN: Best modern file transfer/synchronization protocol?
#25I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well
rclone is garbage if you need performance or just copy between two points you control. it shines when you need to sprinkle your data over many "clouds"
So what do you use?
Re: Ask HN: Best modern file transfer/synchronization protocol?
#26I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well
rclone is no protocol though
Re: Ask HN: Best modern file transfer/synchronization protocol?
#27Earlier quoted context omitted.
I've starting using rclone over rsync for this application. Rclone can do segmented transfers and handles large numbers of files better, at least in my experience.
What are segmented transfers?
Send multiple pieces of the file and reconstruct the file on the other end. It's more reliable to send smaller chunks and the transfer can be hash validated on individual chunks and after reconstruction. Multiple download streams typically performs better too (from single or multiple servers).
e.g. BitTorrent for the multiple servers case.
Re: Ask HN: Best modern file transfer/synchronization protocol?
#28Earlier quoted context omitted.
Thanks - is there a reference implementation for rsync that works on windows and iOS? My impression was it was more of a CLI tool for Linux/macOS.
If you are looking to integrate into another app, maybe check https://librsync.github.io/ out.
Unfortunately one of the caveats about what librsync is not (from the link):
librsync also does not include any network functions for talking to SSH or any other server. To access a remote filesystem, you need to provide your own code or make use of some other virtual filesystem layer.
Having this seems to be one of the primary requirements.
Re: Ask HN: Best modern file transfer/synchronization protocol?
#29No need to get fancy, scp or rsync are the tried and true options here.
Or on Windows, Xcopy.
Re: Ask HN: Best modern file transfer/synchronization protocol?
#30I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well
rclone is garbage if you need performance or just copy between two points you control. it shines when you need to sprinkle your data over many "clouds"
I'd argue the opposite. Rclone's parallelization options are unmatched.