I've often heard that the best way to send large amounts of data is to actually send it via snail mail. But if that is not an option what is the best way to send, say 500GB, over the internet? Past Experience preferred but creative solutions welcome!
Ask HN: How do you handle transferring large files over the internet?
1–10 of 35 posts
Re: Ask HN: How do you handle transferring large files over the internet?
#2rsync
Re: Ask HN: How do you handle transferring large files over the internet?
#3rsync
Indeed. I've been trying all sorts of weird stuff, but this takes the cake. Ubiquitous, rock-solid, sane. Plus, no worrying "is it done yet? Do I have the latest version?" Just let it run (again) - this makes it rather foolproof.
Re: Ask HN: How do you handle transferring large files over the internet?
#4Re: Ask HN: How do you handle transferring large files over the internet?
#5If, for some reason, you are averse to rsync (there are plenty of Windows clients), then bittorrent is second-best: it works just as well for transferring private data, just turn off all the metadata broadcast options and make sure that encryption is required. (Not so easy for incremental transfers, though)
Re: Ask HN: How do you handle transferring large files over the internet?
#6I assume you're transferring between two hosts you both control. Some sane options are:
- rsync
- SFTP-over-SSH, or SFTP over a dedicated VPN
- private, encrypted torrents
Re: Ask HN: How do you handle transferring large files over the internet?
#7I assume you're transferring between two hosts you both control. Some sane options are: - rsync - SFTP-over-SSH, or SFTP over a dedicated VPN - private, encrypted torrents
SFTP gets tricky on resume; rsync and BT have builtin data integrity checks.
Re: Ask HN: How do you handle transferring large files over the internet?
#8I assume you're transferring between two hosts you both control. Some sane options are: - rsync - SFTP-over-SSH, or SFTP over a dedicated VPN - private, encrypted torrents
[deleted]
Re: Ask HN: How do you handle transferring large files over the internet?
#9rsync
Indeed. I've been trying all sorts of weird stuff, but this takes the cake. Ubiquitous, rock-solid, sane. Plus, no worrying "is it done yet? Do I have the latest version?" Just let it run (again) - this makes it rather foolproof.
Does rsync auto-resume after failed connections?
Re: Ask HN: How do you handle transferring large files over the internet?
#10Earlier quoted context omitted.
Indeed. I've been trying all sorts of weird stuff, but this takes the cake. Ubiquitous, rock-solid, sane. Plus, no worrying "is it done yet? Do I have the latest version?" Just let it run (again) - this makes it rather foolproof.
Does rsync auto-resume after failed connections?
I've written a script that every minute tests to see if the appropriate rsync command is running. If not, it simply runs it again. In that way it effectively restarts and gracefully resumes.
This Google search:
https://www.google.co.uk/search?q=rsync+auto-restart+failed+...
returns this link:
http://superuser.com/questions/302842/resume-rsync-over-ssh-...
which contains this script:
#!/bin/bash
while [ 1 ]
do
rsync -avz --partial source dest
if [ "$?" = "0" ] ; then
echo "rsync completed normally"
exit
else
echo "Rsync failure. Backing off and retrying..."
sleep 180
fi
done
When the connection dies, rsync will quit
with a non-zero exit code. This script simply
keeps re-running rsync, letting it continue
until the synchronisation completes normally.