My experience with using cp to copy 432 million files (39 TB)

The later replies regarding the size of the data structures cp is using are also worth reading. This is a case where pushing the command farther can make you think harder about the computations being done.

Re: My experience with using cp to copy 432 million files (39 TB)

#5

Post by 0x0 » Thu, Sep 11, 2014, 10:21 PM UTC

I wonder how well rsync would have fared here.

Re: My experience with using cp to copy 432 million files (39 TB)

#6

Post by mturmon » Thu, Sep 11, 2014, 10:26 PM UTC

gaius wrote: post #2

I would probably have used tar|tar for this, or rsync.

About rsync: if you just use -a, it does not copy hard links correctly:

  -a, --archive
  [...]  Note that -a does not preserve hardlinks, because 
  finding multiply-linked files is expensive. You must 
  separately specify -H.

If you do specify -H, rsync does keep track of hard links, but presumably at the cost of keeping a data structure similar to cp:

  -H, --hard-links
  This tells rsync to look for hard-linked files in the source and 
  link together the corresponding files on the destination. Without 
  this option, hard-linked files in the source are treated as though 
  they were separate files. [...]

Of course, rsync could be more efficient at keeping track of the hard links than cp, but there's no reason to believe a priori that it would be.

Re: My experience with using cp to copy 432 million files (39 TB)

#7

Post by __david__ » Thu, Sep 11, 2014, 10:28 PM UTC

gaius wrote: post #2

I would probably have used tar|tar for this, or rsync.

Rsync has similar issues when you are copying hard links. It has to keep a table of all the inodes/filename pairs it sees so it can detect hard links and relink them on the destination. This can be big if you have a ton of hard links (say in a backup situation).

Rsync at least acts reasonably idempotent, so you can just run it again if it gets interrupted, which is usually why I use it for large copies.

I don't remember off the top of my head if tar handles hard links—it may be inappropriate for this usage.

Re: My experience with using cp to copy 432 million files (39 TB)

#8

Post by kazinator » Thu, Sep 11, 2014, 10:29 PM UTC

mturmon wrote: post #6

gaius wrote: post #2

I would probably have used tar|tar for this, or rsync.

About rsync: if you just use -a, it does not copy hard links correctly: -a, --archive [...] Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H. If you do specify -H, rsync does keep track of hard links, but presumably at the cost of keeping a data structure similar to cp: -H, --hard-links This tells rsync to look for hard-linked files in the so…

Darn, thanks! I've been using rsync for incremental backups (themselves hard-link-based) for years, but neglecting to do this. That is to say, snapshot N+1 shares structure with backup N to save space (via hard links) via --link-dest. But files may be replicated because -H (missing) is orthogonal of --link-dest.

Re: My experience with using cp to copy 432 million files (39 TB)

#9

Post by jewel » Thu, Sep 11, 2014, 10:30 PM UTC

maaku wrote: post #3

Another lesson: routinely scrub your RAID arrays.

On debian-based systems, /etc/cron.d/mdadm will already do this on the first Sunday of the month.

Re: My experience with using cp to copy 432 million files (39 TB)

#10

Post by IvyMike » Thu, Sep 11, 2014, 10:31 PM UTC

Interesting.

In Windows-land, the default copy is pretty anemic, so probably most people avoid it for serious work.

I'd probably use robocopy from the command line. And if I was being lazy, I'd use the Teracopy GUI.

I think my limit for a single copy command has been around 4TB with robocopy--and that was a bunch of large media files, instead of smaller more numerous files. Maybe there's a limit I haven't hit.