Ask HN: Best modern file transfer/synchronization protocol?

#1

Post by daveidol » Fri, Dec 22, 2023, 5:46 PM UTC

Hi HN,

I'm looking to build something to transfer files between two devices on a network (one client and one server, both of which are under my control).

Obviously I could write something bespoke with raw TCP sockets, but rather than reinventing the wheel I'm curious about what existing options people recommend. I assume there are some better technologies than FTP nowadays?

Ideally some kind of built-in fault tolerance would be great, because my plan is to use this on a phone/in an environment where the connection could be interrupted.

Edit: just to clarify - this is something I want to build into an application I am writing, ideally with support across iOS (client), Windows, and mac (server).

One way transfer is all I need, and I mostly plan on photos/videos (so multiple files ~3-20MB in size).

Thanks!

Re: Ask HN: Best modern file transfer/synchronization protocol?

#2

Post by jayknight » Fri, Dec 22, 2023, 5:53 PM UTC

I would probably start with rsync.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#3

Post by sneak » Fri, Dec 22, 2023, 5:53 PM UTC

rsync over ssh for one-shots.

syncthing for continuous use.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#4

Post by bhaney » Fri, Dec 22, 2023, 5:55 PM UTC

Yeah I usually just use rsync for this. In a loop if the network is unreliable.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#5

Post by whalesalad » Fri, Dec 22, 2023, 5:55 PM UTC

seconding rsync and syncthing.

the server could expose an smb or nfs share, the client could mount it, and then sync to that mount.

rsync over ssh also works, if you do not want to run smb/nfs.

this is also a cool tool https://rclone.org/

Re: Ask HN: Best modern file transfer/synchronization protocol?

#6

Post by thedaly » Fri, Dec 22, 2023, 5:58 PM UTC

sneak wrote: post #3

rsync over ssh for one-shots. syncthing for continuous use.

I've starting using rclone over rsync for this application. Rclone can do segmented transfers and handles large numbers of files better, at least in my experience.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#7

Post by daveidol » Fri, Dec 22, 2023, 6:02 PM UTC

jayknight wrote: post #2

I would probably start with rsync.

Thanks - is there a reference implementation for rsync that works on windows and iOS? My impression was it was more of a CLI tool for Linux/macOS.

Re: Ask HN: Best modern file transfer/synchronization protocol?

#8

Post by sneak » Fri, Dec 22, 2023, 6:05 PM UTC

thedaly wrote: post #6

sneak wrote: post #3

rsync over ssh for one-shots. syncthing for continuous use.

I've starting using rclone over rsync for this application. Rclone can do segmented transfers and handles large numbers of files better, at least in my experience.

What are segmented transfers?

Re: Ask HN: Best modern file transfer/synchronization protocol?

#9

Post by xnx » Fri, Dec 22, 2023, 6:06 PM UTC

thedaly wrote: post #6

sneak wrote: post #3

rsync over ssh for one-shots. syncthing for continuous use.

I've starting using rclone over rsync for this application. Rclone can do segmented transfers and handles large numbers of files better, at least in my experience.

rclone is great! So useful as an alternative/superior client for cloud storage too (Google Drive, OneDrive, etc.)

Re: Ask HN: Best modern file transfer/synchronization protocol?

#10

Post by necovek » Fri, Dec 22, 2023, 6:14 PM UTC

daveidol wrote: post #7

jayknight wrote: post #2

I would probably start with rsync.

Thanks - is there a reference implementation for rsync that works on windows and iOS? My impression was it was more of a CLI tool for Linux/macOS.

If you are looking to integrate into another app, maybe check https://librsync.github.io/ out.