Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

221–230 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#221

Post by taeric »

jayvanguard wrote: post #112
breadbox wrote: post #110

Earlier quoted context omitted.

At one time there was; it was called the Internet. The archive still exists, but it's been made harder to browse through due to being jumbled up with javascript and cat gifs.

It's true. We should have never let the public on the Internet. It has been downhill since then.

False choice, isn't it? I mean, the complaint isn't that the public now has sites with massive javascript and related technologies. The complaint is that it has muscled out useful sites that did not use those technologies. And it should be heavily noted that the heavy muscles that have pushed out many of these sites is not necessarily "the public."

Re: My experience with using cp to copy 432 million files (39 TB)

#222

Post by pflanze »

I've written a program that attempts to deal with the given situation gracefully: instead of using a hash table, it creates a temporary file with a list of inode/device/path entries, then sorts this according to inode/device, then uses the sorted list to perform the copying/hardlinking. The idea is that sorting should work well with much lower RAM requirements than the size of the file to be sorted (due to data locality, unless the random accesses with the hash, it will be able to work with big chunks, at least when done right (a bit hand-wavy, I know, this is called an "online algorithm" and I remember Knuth having written about those, haven't had the chance to recheck yet); the program is using the system sort command, which is hopefully implementing this well already).

The program stupidly calls "cp" right now for every individual file copy (not the hard linking), just to get the script done quickly, it's easy to replace that with something that saves the fork/exec overhead; even so, it might be faster than the swapping hash table if the swap is on a spinning disk. Also read the notes in the --help text. I.e. this is a work in progress as a basis to test the idea, it will be easy to round off the corners if there's interest.

https://github.com/pflanze/megacopy

PS. the idea of this is to make copying work well with the given situation on a single machine, unless the approach taken by the dcp program mentioned by fintler which seems to rely on a cluster of machines.

There may also be some more discussion about this on the mailing list: http://lists.gnu.org/archive/html/coreutils/2014-09/msg00013...

Re: My experience with using cp to copy 432 million files (39 TB)

#223

Post by grondilu »

brazzy wrote: post #204
grondilu wrote: post #190

Earlier quoted context omitted.

I doubt it, since you're using a pipe. It's not like you're actually storing the whole archive anywhere. Also, and just to be clear, I did not invent this idea of using tar for this. I remember having read it in a Unix manual when I was learning about this OS. And even if using tar is not a good idea, it should certainly be considered and as such I don't understand why OP doesn't even mention it in his post.

The pipe is not the issue; you actually are storing the full paths of all files in the archive somewhere (namely in RAM) so that you're able to decide whether a given file you're looking at is an alternative hardlink for a file you have already sent. Yes, using tar for copying is not new and has its use cases, but this is not one of them.

Certainly not for all files.

The filesystem keeps a track of the number of nodes links for each file. So tar does not have to memorize the inode for each file, but only for those who have duplicates. So unless a large amount of files have more than one hard link, this is probably not a problem.

We will never know though, considering the author did not try it. Neither did he explain why he did not, despite the fact that it is the recommended Unix procedure.

Re: My experience with using cp to copy 432 million files (39 TB)

#224

Post by rimantas »

spydum wrote: post #196
KaiserPro wrote: post #179

Earlier quoted context omitted.

WE use 10 rsyncs in parallel to copy .5pb in less than 3 days. CPU is cheap.

Have done the same, found that rsync seems to not natively parallelize itself, so spread across 20 cores, it really screamed.

Damn, I could use some advice there. Right now I have a system which uses rsync to backup some files. It takes about 8 hours to go over 7+ million files (~3.5TB). I'd love to speed up this. I should mention that copy is done over curlftpfs :(.

Re: My experience with using cp to copy 432 million files (39 TB)

#225

Post by Eiriksmal »

cnvogel wrote: post #187
dbbolton wrote: post #13

>We use XFS Why?

I personally still consider XFS a very mature and reliable filesystem. Both in terms of utility programs and kernel implementation. If I remember correctly, it was ported to linux from SGI/Irix where it was used for decades. It also was the default fs for RedHat/centos for a long time, so it might still have stuck at many shops. Heres my anecdotal datapoint on which I base my personal believe: From about 10-6 years a…

As a blithe, new-Linux user (3.5 years), I was bumfuzzled when I saw RHEL/CentOS 7 switched from ext4 to XFS, figuring it to be some young upstart stealing the crown from the king. Then I did some Googling and figured out that XFS is as old as ext2! I'm looking forward to discovering how tools like xfs* can make my life easier.

Re: My experience with using cp to copy 432 million files (39 TB)

#226

Post by aidenn0 »

click170 wrote: post #99
greenknight wrote: post #81

Earlier quoted context omitted.

I had a RAID 6 have a failed disk a couple weeks ago... 8 x 3TB drives... 18TB usable. Took 7 hours to rebuild. Disks were connected via SATA 3 and was using mdadm

In my experience working with Debian, the rebuild is sensitive to disk usage. If you start using the disks, the rebuild will slow down intentionally so as to not choke the drives. This could explain why it was fast for you but not for the parent. Edit: To clarify, I was using mdadm, not hardware raid.

I you can tune the speed when the disks are in use; I usually bump it to around 60MB/s as long as there isn't a live database on the disk; the default is something insanely slow, and it impacts read/writes almost as much as the higher value.

Re: My experience with using cp to copy 432 million files (39 TB)

#227

Post by pflanze »

pflanze wrote: post #123
mrb wrote: post #114

Earlier quoted context omitted.

In fact, 17e9/432e6=39.35 B/file, pretty spot on. Seems like there's some room for optimization It does not suggest there is a possible optimization. cp needs to store the pointer and the path itself in memory, so it is more than 8 bytes. ~39 bytes/file means the average path length was 15 bytes (8-byte source device + 8-byte source inode + 8-byte pointer + ~15-byte path = ~39 bytes). I guess one theoretical optimiza…

Yes, you're right, I made a jump in my thinking; I wanted to make the point that you could exclude the storage for the paths from the working set size, if there is only rarely a match. Problem is we don't have a number for the working set size (except that it must be between 10 GB and 17 GB), and my brain just used the 17 GB. But your calculation is missing the empty space in the hash table. When using open addressin…

> Edit 2: a different solution

See "megacopy" in my top level comment.

Re: My experience with using cp to copy 432 million files (39 TB)

#228

Post by dredmorbius »

aruggirello wrote: post #161
lamontcg wrote: post #104

Earlier quoted context omitted.

if you want to copy everything and there's nothing at the target: rsync --whole-file --ignore-times that should turn off the metadata checks and the rsync block checksum algorithm entirely and transfer all of the bits at the source to the dest without any rsync CPU penalty. also for this purpose it looks like -H is also required to preserve hard links which the man page notes: "Note that -a does not preserve hardlink…

You cannot exaggerate what rsync can do... I use it to backup my almost full 120gb Kubuntu system disk daily, and it goes through 700k files in ~ 2-3 minutes. Oh, and it does it live, while I'm working.

To be fair: the case described here is two magnitudes larger in both directions than your use case.

And the critical aspect from a performance perspective was where the hash table became too large to fit into memory. Performance of all sorts goes pear-shaped when that happens.

rsync is pretty incredible, but, well, quantity has a quality all its own, and the scale involved here (plus the possible in-process disk failure) likely wasn't helping much.

Re: My experience with using cp to copy 432 million files (39 TB)

#229

Post by dredmorbius »

tedunangst wrote: post #213
calvins wrote: post #207

I would usually use the tarpipe mentioned already by others for this sort of thing (although I probably wouldn't do 432 million files in one shot): (cd $SOURCE && tar cf - .) | (mkdir -p $DEST && cd $DEST && tar xf -) Another option which I just learned about through reading some links from this thread is pax ( http://en.wikipedia.org/wiki/Pax_%28Unix%29 ), which can do it with just a single process: (mkdir -p $DEST…

You know how tar handles hardlinks, right? By creating a giant hash table of every file.

How's that going to scale with memory? In-memory hash tables were the downfall of cp here.

Re: My experience with using cp to copy 432 million files (39 TB)

#230

Post by dredmorbius »

fintler wrote: post #93

I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/h…

NB, that PDF seems to have a number of formatting glitches, e.g., the first sentence: "The amount of scienti c data". Numerous others as well, under both xpdf and evince.
Post reply on HN