Ask HN: How do you handle transferring large files over the internet? • Hacker News Board

Re: Ask HN: How do you handle transferring large files over the internet?

#31

Post by Piskvorrr » Mon, Jul 11, 2016, 7:22 AM UTC

atsaloli wrote: post #27

kraemate wrote: post #9

Earlier quoted context omitted.

Does rsync auto-resume after failed connections?

Sure, just use --partial switch: --partial keep partially transferred files

That is incorrect. This will keep partial transfers in destination (and when invoked again, will continue tranferring from that point onwards), but will not restart an aborted transfer (e.g. for a broken connection).

In other words, this feature is a prerequisite for auto-resume, not auto-resume itself (which can be scripted in ~10 lines, as shown in a sibling thread).

Re: Ask HN: How do you handle transferring large files over the internet?

#32

Post by Piskvorrr » Mon, Jul 11, 2016, 9:00 AM UTC

soulbadguy wrote: post #30

Do you need to transmit 500GB every time or just a diff from a previous dataset ? If it's the later case, using the send/receive functionality of a file system with snapshot and incremental backup (ZFS,BTRFS etc..) can be significantly faster than using pure rsync. Rsync would needs to scan the complete 500GB of data to find the blocks to send, while send/receive can compute the diff much faster

Negligible W/R/T transfer time. While using native capabilities of ZFS is awesome, you're now locked into a particular FS at both sides of the transfer (this may or may not be an issue).

(Also, there's a patch for rsync that allows you to force computing the checksum in batch, not on each invocation; but that's getting into hairy territory that's rarely needed - I used it in exactly one case so far)

Re: Ask HN: How do you handle transferring large files over the internet?

#33

Post by atsaloli » Mon, Jul 11, 2016, 3:39 PM UTC

Piskvorrr wrote: post #29

atsaloli wrote: post #28

Earlier quoted context omitted.

About 9 years ago, while working as a system administration consultant, I had a gig to fly a portable hard drive with about 360 GB from LA to St. Louis as part of a migration of a web application. It was faster than the network connections available to my client at the time. I remember calculating the throughput... I asked why don't you just FedEx it? It's too important, the client said, and we know and trust you. It…

Well...you are probably not going to throw a hard drive at a client's door and run. A delivery guy might, as it's just another cardboard package, not priceless data (might be easier now with SSDs). Indeed, getting a trustworthy courier service is so hard that actually sending an in-house employee is worthwhile, even though their hourly rates make this extremely expensive: you are removing tens of abstraction layers,…

Fair enough. :)

Re: Ask HN: How do you handle transferring large files over the internet?

#34

Post by atsaloli » Thu, Jul 14, 2016, 1:16 PM UTC

Piskvorrr wrote: post #31

atsaloli wrote: post #27

Earlier quoted context omitted.

Sure, just use --partial switch: --partial keep partially transferred files

That is incorrect. This will keep partial transfers in destination (and when invoked again, will continue tranferring from that point onwards), but will not restart an aborted transfer (e.g. for a broken connection). In other words, this feature is a prerequisite for auto-resume, not auto-resume itself (which can be scripted in ~10 lines, as shown in a sibling thread).

Thanks! But no need to script, you can just use

            --append                append data onto shorter files

to resume interrupted transfers.

Re: Ask HN: How do you handle transferring large files over the internet?

#35

Post by atsaloli » Thu, Jul 14, 2016, 1:19 PM UTC

kraemate wrote: post #9

Piskvorrr wrote: post #3

Earlier quoted context omitted.

Indeed. I've been trying all sorts of weird stuff, but this takes the cake. Ubiquitous, rock-solid, sane. Plus, no worrying "is it done yet? Do I have the latest version?" Just let it run (again) - this makes it rather foolproof.

Does rsync auto-resume after failed connections?

Yes, it it does, if you use:

   --partial               keep partially transferred files

   --append                append data onto shorter files