Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you handle transferring large files over the internet?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: Ask HN: How do you handle transferring large files over the internet?

#11

Post by Piskvorrr »

kraemate wrote: post #9
Piskvorrr wrote: post #3

Earlier quoted context omitted.

Indeed. I've been trying all sorts of weird stuff, but this takes the cake. Ubiquitous, rock-solid, sane. Plus, no worrying "is it done yet? Do I have the latest version?" Just let it run (again) - this makes it rather foolproof.

Does rsync auto-resume after failed connections?

You mean, if the connection fails before completing? I don't think so (I believe that's a feature).

For automated transfers ("retry until done"), I use the lsyncd wrapper (which also watches the source files, so you don't need to poll for changes: it wakes up by inotify).

Re: Ask HN: How do you handle transferring large files over the internet?

#12

Post by lucasch »

Piskvorrr wrote: post #5

If, for some reason, you are averse to rsync (there are plenty of Windows clients), then bittorrent is second-best: it works just as well for transferring private data, just turn off all the metadata broadcast options and make sure that encryption is required. (Not so easy for incremental transfers, though)

We considered rsync but were wondering if there were more specialized tools available. We figured that those who work in the scientific community would have a way to transfer their large data sets between institutions.

Re: Ask HN: How do you handle transferring large files over the internet?

#13

Post by kraemate »

CarolineW wrote: post #10
kraemate wrote: post #9

Earlier quoted context omitted.

Does rsync auto-resume after failed connections?

I've written a script that every minute tests to see if the appropriate rsync command is running. If not, it simply runs it again. In that way it effectively restarts and gracefully resumes. This Google search: https://www.google.co.uk/search?q=rsync+auto-restart+failed+... returns this link: http://superuser.com/questions/302842/resume-rsync-over-ssh-... which contains this script: #!/bin/bash while [ 1 ] do rsync -…

Wow, thanks for the script. Surprising in its simplicity. I would have thought this use-case was popular enough to warrant specialized tools etc. Especially in the scientific community where they transfer large files.

Re: Ask HN: How do you handle transferring large files over the internet?

#14

Post by ColinWright »

lucasch wrote: post #12
Piskvorrr wrote: post #5

If, for some reason, you are averse to rsync (there are plenty of Windows clients), then bittorrent is second-best: it works just as well for transferring private data, just turn off all the metadata broadcast options and make sure that encryption is required. (Not so easy for incremental transfers, though)

We considered rsync but were wondering if there were more specialized tools available. We figured that those who work in the scientific community would have a way to transfer their large data sets between institutions.

We transfer large files containing raw radar data, and moderate sized files contains databases of target movements and track information.

We use rsync.

When I worked for the local university we had to transfer data between machines to run experimental parallel programs on so-called "big data."

We used rsync.

Re: Ask HN: How do you handle transferring large files over the internet?

#15

Post by CarolineW »

kraemate wrote: post #13
CarolineW wrote: post #10

Earlier quoted context omitted.

I've written a script that every minute tests to see if the appropriate rsync command is running. If not, it simply runs it again. In that way it effectively restarts and gracefully resumes. This Google search: https://www.google.co.uk/search?q=rsync+auto-restart+failed+... returns this link: http://superuser.com/questions/302842/resume-rsync-over-ssh-... which contains this script: #!/bin/bash while [ 1 ] do rsync -…

Wow, thanks for the script. Surprising in its simplicity. I would have thought this use-case was popular enough to warrant specialized tools etc. Especially in the scientific community where they transfer large files.

It's simple enough that it's the sort of thing I type out in 30 seconds and there it is. No need for specialist tools - finding the tool, remembering how to use it, working out the right parameters ...

Easier, faster, and more flexible just to write the script. It's what I do.

Re: Ask HN: How do you handle transferring large files over the internet?

#16

Post by lucasch »

ColinWright wrote: post #14
lucasch wrote: post #12

Earlier quoted context omitted.

We considered rsync but were wondering if there were more specialized tools available. We figured that those who work in the scientific community would have a way to transfer their large data sets between institutions.

We transfer large files containing raw radar data, and moderate sized files contains databases of target movements and track information. We use rsync. When I worked for the local university we had to transfer data between machines to run experimental parallel programs on so-called "big data." We used rsync.

Ahh ok thats good to hear. Have you considered using multipath transport protocols with something like rsync? I am curious if it could benefit this situation. MPTCP sounds like an interesting protocol if you control both hosts. https://www.multipath-tcp.org/

Re: Ask HN: How do you handle transferring large files over the internet?

#17

Post by teddyc »

I would use scp with the -C option to compress the data for the first attempt to transfer the file. Any subsequent attempts to transfer the file should use rsync (this includes any updates to the file).

Another option is to use a cloud data storage provider (Dropbox, Box, Google Drive, etc) and install their software that keeps files in sync. Then you can just put the file in a local folder and let their software sync it to the cloud. After sync is complete, you email a link to share the file. Of course, their software is probably just a GUI for rsync.

Re: Ask HN: How do you handle transferring large files over the internet?

#18

Post by ColinWright »

lucasch wrote: post #16
ColinWright wrote: post #14

Earlier quoted context omitted.

We transfer large files containing raw radar data, and moderate sized files contains databases of target movements and track information. We use rsync. When I worked for the local university we had to transfer data between machines to run experimental parallel programs on so-called "big data." We used rsync.

Ahh ok thats good to hear. Have you considered using multipath transport protocols with something like rsync? I am curious if it could benefit this situation. MPTCP sounds like an interesting protocol if you control both hosts. https://www.multipath-tcp.org/

We were/are always restricted by intermediate limits on throughput, so it's never been useful or interesting to consider alternatives.

YMMV, but if you want to improve throughput, consider carefully where your data has to go through. But rsync is rock-solid, well-understood, mature, and just does exactly what it is intended to do.

Re: Ask HN: How do you handle transferring large files over the internet?

#19

Post by Piskvorrr »

teddyc wrote: post #17

I would use scp with the -C option to compress the data for the first attempt to transfer the file. Any subsequent attempts to transfer the file should use rsync (this includes any updates to the file). Another option is to use a cloud data storage provider (Dropbox, Box, Google Drive, etc) and install their software that keeps files in sync. Then you can just put the file in a local folder and let their software syn…

Why not use rsync for the first transfer as well? There's `-z` for the same effect...

As for commercial providers - I've seen rate limits (never did it saturate the link) and size limits (0.5 TB will cost you extra). Moreover, the data will go through an intermediate hop, which is slow (Dropbox starts downloading only when upload completes, doubling your transfer time), plus your data is now...somewhere (which could be a regulatory issue, depending on the data).

Post reply on HN