Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Out-Tridging Tridge by improving rsync

anchor.com.au

Post reply on HN
Print view View original

11–15 of 15 posts

Re: Out-Tridging Tridge by improving rsync

#11

Post by bigiain »

chetanahuja wrote: post #2

ok this article had me intrigued (and the fix presented to speedup rsync for large files is good) but then, this gem: "The thing is, modern networks aren’t like that at all, they’re high bandwidth and low latency" No they're not. Unless you mean two machines sitting side by side in a data center or perhaps within the same metro area connected via wired connections. Two machines sitting on either coast of US with the…

And for some of us, this is a typical experience:

  25 packets transmitted, 25 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 263.361/321.076/375.393/29.289 ms
(That's from Sydney Australia to a DigitalOcean Droplet in nyc1)

Re: Out-Tridging Tridge by improving rsync

#12

Post by keeperofdakeys »

kbenson wrote: post #3

does copying/moving a file over another not trigger copy-on-write in btrfs? If not, it seems a much simpler (but much less cool and useful for all) solution would be to patch rsync with an option to allow writing the temp file over the original when done. While still non-atomic, you'll get the copy-on-write semantics you need. Unfortunately it will use much more IO. There are ways to mitigate the extra IO, such as cr…

Most Unix file systems don't have the semantics of a "move". You unlink an inode from a filename, and link another inode (usually the inode for the tmp file). Then you unlink the original tmp filename. As far as btrfs is concerned, there is no relation between these inodes, and without copying the file (like you suggest), you can't improve this.

Re: Out-Tridging Tridge by improving rsync

#13

Post by mcpherrinm »

chetanahuja wrote: post #2

ok this article had me intrigued (and the fix presented to speedup rsync for large files is good) but then, this gem: "The thing is, modern networks aren’t like that at all, they’re high bandwidth and low latency" No they're not. Unless you mean two machines sitting side by side in a data center or perhaps within the same metro area connected via wired connections. Two machines sitting on either coast of US with the…

In the article, the author mentions sending "1.2 terabytes in a few hours" with 10ms latency. This sounds like a gigabit network with a few router hops in the middle. Maybe 100mbit if "a few hours" is interpreted longer than I would. So we're talking about connecting two machines, possibly in different buildings, but likely within a thousand kilometers.

It's honestly the use case I have the most often, and certainly one useful to have tools supporting the use case. This person seems to be worried about off-site backups, so I'm thinking "enterprise", not "cross-country home user".

Re: Out-Tridging Tridge by improving rsync

#15

Post by kbuck »

fhars wrote: post #9
gmac wrote: post #8

Sounds like a worthwhile improvement to rsync, but I wonder why this setup is preferred to duplicity [1] or rdiff-backup [2], which both also use rsync (librsync) to perform incremental backups. I've had good experiences with duplicity in particular. [1] http://duplicity.nongnu.org/ [2] http://www.nongnu.org/rdiff-backup/

rdiff-bachup will give you copies of the changed files (so 1.2TiB for the database file mentioned in the article) every time you run a backup for each old version you keep. It will not transfer that much data over the network (since it uses the rsync algorithm), but it stores that much on disk. On the other hand, if you use a filesystem with copy-on-write snapshots and in-place modification of the changed files, you…

This isn't entirely correct; rdiff-backup will give you a full copy of the latest version of the file as well as a set of binary diffs that can be applied in sequence to roll it back to an earlier version. rdiff-backup will actually end up being a little more space efficient for each incremental change since its diffs don't need to store entire filesystem blocks.
Post reply on HN