Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

161–170 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#161

Post by aruggirello »

lamontcg wrote: post #104
golan wrote: post #89

Earlier quoted context omitted.

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

if you want to copy everything and there's nothing at the target: rsync --whole-file --ignore-times that should turn off the metadata checks and the rsync block checksum algorithm entirely and transfer all of the bits at the source to the dest without any rsync CPU penalty. also for this purpose it looks like -H is also required to preserve hard links which the man page notes: "Note that -a does not preserve hardlink…

You cannot exaggerate what rsync can do... I use it to backup my almost full 120gb Kubuntu system disk daily, and it goes through 700k files in ~ 2-3 minutes. Oh, and it does it live, while I'm working.

Re: My experience with using cp to copy 432 million files (39 TB)

#162

Post by jwr »

gaius wrote: post #2

I would probably have used tar|tar for this, or rsync.

This. I guess anyone who began using UNIX systems back in the SunOS days or earlier would never think of using cp for this job. Well, mostly because the vendor-provided cp usually sucked badly (no recursive copy), but also because tar has always been such a great, reliable tool.

The only disadvantage of tar is its peculiar syntax, but that's something you get used to.

Re: My experience with using cp to copy 432 million files (39 TB)

#163

Post by halayli »

fintler wrote: post #93

I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/h…

couple nitpicks:

Check return value of malloc

You don't need \ when breaking function parameters

Re: My experience with using cp to copy 432 million files (39 TB)

#164

Post by rbh42 »

thaumaturgy wrote: post #106
hnha wrote: post #29

Earlier quoted context omitted.

What's the benefit of using a tarpipe locally?

tarpipes can handle gnarly piles of hardlinked files without sweating. BackupPC is free software for network backups that builds a massively hardlinked file pool; tarpipes are the only thing that can reliably copy BackupPC pools from one location to another.

Not so. Both the sending and the receiving tar processes will need a data structure keeping track of which inodes they've already processed. They can skip inodes with a link count of 1, but if all inodes have multiple links (as in this case), the overhead will be twice that of a single cp process.

Re: My experience with using cp to copy 432 million files (39 TB)

#165

Post by 72deluxe »

squiguy7 wrote: post #140
breadbox wrote: post #110

Earlier quoted context omitted.

At one time there was; it was called the Internet. The archive still exists, but it's been made harder to browse through due to being jumbled up with javascript and cat gifs.

"due to being jumbled with javascript..." - This made my day.

It's also an astute observation that is incredibly true! Most sites now incorporate JavaScript.

I remember when it was "new-fangled" (and DHTML anyone?) and everyone put those annoying cursor trackers that trailed blobs from where your cursor was.

Re: My experience with using cp to copy 432 million files (39 TB)

#166

Post by brazzy »

makmanalp wrote: post #94
brazzy wrote: post #58

Earlier quoted context omitted.

No giant bookkeeping datastructures that end up with the process thrashing in swap, because tar is designed to work with so many files that you don't want to keep the metadata in memory.

This is true if the bookkeeping was irrelevant and unnecessary. I imagine the scenario where your copy gets stopped in the middle is worse when you use the tar method.

It's not as if cp gives you anything more useful in that case. The bookkeeping that caused the problem was a hashtable of files that have been copied, necessary to handle hard links. Would be interesting how tar does that; if it can't do any better then the root comment and mine above are wrong.

Re: My experience with using cp to copy 432 million files (39 TB)

#167

Post by [deleted] »

Rapzid wrote: post #124
angry_octet wrote: post #96

Earlier quoted context omitted.

I was simplifying... dump backs up inodes not blocks. Some inodes point to file data and some point to directory data. Hard links are references to the same inode in multiple directory entries, so when you run xfsrestore, the link count increments as the FS hierarchy is restored. xfsdump/zfs send are file system aware, unlike dd, and can detect fs corruption (ZFS especially having extensive checksums). In fact, any i…

If the risk of keeping the system running while the array rebuilt was deemed to high, I would have just gone with a dd/ddrescue of the remaining disks onto new disks and then moved on from there. +1 for mentioning ZFS. It's really quite amazing. Almost like futuristic alien technology compared to the other freely available file systems.

[deleted]

Re: My experience with using cp to copy 432 million files (39 TB)

#168

Post by brazzy »

grondilu wrote: post #143

On Unix, isn't it considered bad practice to use cp in order to copy a large directory tree? IIRC, the use of tar is recommended. Something like: $ (cd $origin && tar cf - *) | (cd $destination && tar xvf - )

Would have been even worse in this case due to the requirement to preserve hard links.

Re: My experience with using cp to copy 432 million files (39 TB)

#169

Post by lysium »

golan wrote: post #89
ntsb wrote: post #67

Earlier quoted context omitted.

Wouldn't rsync of been a better and more reliable choice for this?

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

> The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file.

- In this scenario, the receiving side is empty, so there is no need to check every single file.

- Since 3.0, rsync walks the dirtree while copying (unless you use some special options). So, in some sense, rsync is already "deep syncing inside the FS tree", as you put it.

Re: My experience with using cp to copy 432 million files (39 TB)

#170

Post by lysium »

thrownaway2424 wrote: post #138
voltagex_ wrote: post #135

Earlier quoted context omitted.

Your post has no information as to why you'd use cpio.

OK, fair point. cpio doesn't have global data structures that grow with the size of the operation, which cp does have. rsync prior to 3.0 also had to scan the entire job before it would begin the transfer, which can take forever. cpio operates on a stream, in the form of find | cpio , so cpio only sees one file from the stream at a time. Find, at least GNU find, is also written such that its resources don't vary with…

Does your setup preserve hard links? Because this is from where OP's problems originated.
Post reply on HN