Ask HN: Best modern file transfer/synchronization protocol? • Hacker News Board

Re: Ask HN: Best modern file transfer/synchronization protocol?

#11

Post by teddyh » Fri, Dec 22, 2023, 6:15 PM UTC

daveidol wrote: post #7

jayknight wrote: post #2

I would probably start with rsync.

Thanks - is there a reference implementation for rsync that works on windows and iOS? My impression was it was more of a CLI tool for Linux/macOS.

For Windows: https://community.chocolatey.org/packages/rsync>

Re: Ask HN: Best modern file transfer/synchronization protocol?

#12

Post by ryukoposting » Fri, Dec 22, 2023, 6:16 PM UTC

I use syncthing for this. It's a little fiddly to set up, and transfer speeds aren't great. However, it's very reliable once configured, and it barely uses any resources after an initial scan of the folder you want to sync.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#13

Post by orbz » Fri, Dec 22, 2023, 6:17 PM UTC

No need to get fancy, scp or rsync are the tried and true options here.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#14

Post by toomim » Fri, Dec 22, 2023, 6:21 PM UTC

Use HTTP. For fault tolerance, use resumeable downloads or resumeable uploads. There is work at the IETF on resumeable uploads right now: https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumabl...

Re: Ask HN: Best modern file transfer/synchronization protocol?

#15

Post by mynegation » Fri, Dec 22, 2023, 6:29 PM UTC

What are your latency and bandwidth requirements? How big are the files? If you are already looking past obvious TCP-based choices like HTTP and FTP, you might be interested in FASP/Aspera https://en.m.wikipedia.org/wiki/Fast_and_Secure_Protocol

Edit: I’ll leave it here just in case it is useful for others but it may or may not be embeddable into your app, especially on the phone.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#16

Post by Flimm » Fri, Dec 22, 2023, 6:31 PM UTC

toomim wrote: post #14

Use HTTP. For fault tolerance, use resumeable downloads or resumeable uploads. There is work at the IETF on resumeable uploads right now: https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumabl...

I've had terrible experiences downloading large files over HTTP. I'm not sure why, range requests don't seem to be reliable or well supported. Something like BitTorrent is much better for large files: it divides the large file into chunks and hashes each chunk, and by default the chunks are downloaded in random order. BitTorrent seems much more reliable than range requests.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#17

Post by oschrenk » Fri, Dec 22, 2023, 6:38 PM UTC

If it’s one way (that wasn’t quite clear from the requirements to me).

take a look at https://tus.io/

Re: Ask HN: Best modern file transfer/synchronization protocol?

#18

Post by rmorey » Fri, Dec 22, 2023, 6:38 PM UTC

I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well

Re: Ask HN: Best modern file transfer/synchronization protocol?

#19

Post by rrix2 » Fri, Dec 22, 2023, 6:42 PM UTC

I built something on top of the Syncthing API this week after using it on its own for years.

A local instance of Syncthing can behave as a robust sync tool + inotify API for applications consuming the files: https://docs.syncthing.net/rest/events-get.html#get-rest-eve...

i believe there's an embeddable golang library, but if you want something easy to use on android check in on syncthing-fork which lets you define more granular sync conditions including "just turn on 5 minutes every hour" https://github.com/Catfriend1/syncthing-android

Re: Ask HN: Best modern file transfer/synchronization protocol?

#20

Post by tetris11 » Fri, Dec 22, 2023, 6:45 PM UTC

rmorey wrote: post #18

I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well

it even has a decent Android client in F-droid, RCX. I just wish it supported SSHFS