Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

151–160 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#151

Post by perlgeek »

pedrocr wrote: post #61

How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all…

This has the slight flaw that you create two copies instead of one for each file (double write IO, double number of inodes) if there aren't many hardlinks.

I don't think that's an acceptable perfomance penalty in most situations.

Re: My experience with using cp to copy 432 million files (39 TB)

#152

Post by ygra »

RyJones wrote: post #39
IvyMike wrote: post #10

Interesting. In Windows-land, the default copy is pretty anemic, so probably most people avoid it for serious work. I'd probably use robocopy from the command line. And if I was being lazy, I'd use the Teracopy GUI. I think my limit for a single copy command has been around 4TB with robocopy--and that was a bunch of large media files, instead of smaller more numerous files. Maybe there's a limit I haven't hit.

Once you get above about a million files on NTFS, life becomes very slow. If you use a DVCS and check out a bunch of projects with a bunch of history, you can easily get to a point where you can't do too much with the drive. This was true as of 7/2008, perhaps 8/2012 have different/better tunings.

Number of files on the drive shouldn't matter that much, actually. In a single folder is a different matter, of course (for that you definitely want to disable 8.3 compatibility). But the only way I can imagine right now where the number of files on the drive makes a difference would be where you crammed the drive almost full, resulting in a very fragmented MFT (normally the file system tries to keep space for growing the MFT, but not if you try to fill that space with data, too).

Re: My experience with using cp to copy 432 million files (39 TB)

#154

Post by theworst »

ori_b wrote: post #136
derefr wrote: post #101

Earlier quoted context omitted.

In effect, hard links (of mutable files) are a declaration that certain files have the same "identity." You can't get this with plain Venti-on-Fossil, but it's a problem with Fossil (objects are immutable), not with Venti. Venti-on-Venti-on-Fossil would work, though, since Venti just creates imaginary files that inherit their IO semantics from their underlying store, and this should apply recursively: 1. create two n…

You seem to be confusing venti and fossil.

Can you explain further? I am not a plan9 expert, by any means, but I'm stuck at where GP made the confusion. Thanks!

Re: My experience with using cp to copy 432 million files (39 TB)

#155

Post by grondilu »

dmckeon wrote: post #147
grondilu wrote: post #143

On Unix, isn't it considered bad practice to use cp in order to copy a large directory tree? IIRC, the use of tar is recommended. Something like: $ (cd $origin && tar cf - *) | (cd $destination && tar xvf - )

Use && there, not ; - consider the result if either of the cd commands fails.

fixed

Re: My experience with using cp to copy 432 million files (39 TB)

#156

Post by noinsight »

gizmo686 wrote: post #68
noinsight wrote: post #42

Earlier quoted context omitted.

> Teracopy I've used FastCopy for GUI based larger transfers, it's open source and can handle larger datasets well in my experience. It also doesn't choke on >MAX_PATH paths. Haven't had problems with it. Supposedly it's the fastest tool around... The only slight issue is that the author is Japanese so the English translations aren't perfect plus the comments in the source are in Japanese.

">MAX_PATH paths" How does this happen?

It happens on pretty much every network share, especially when you address it with an UNC path ("\\server\share\...") instead of mapping it as a drive.

If you have non-Windows users using the share it's pretty much guaranteed, since they don't have a problem with it - only Windows chokes on this.

Even Windows users manage to somehow create such paths (files) occasionally. Then you need to break out cmd.exe or somesuch because Explorer can't touch them.

Re: My experience with using cp to copy 432 million files (39 TB)

#157

Post by matt-attack »

lamontcg wrote: post #104
golan wrote: post #89

Earlier quoted context omitted.

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

if you want to copy everything and there's nothing at the target: rsync --whole-file --ignore-times that should turn off the metadata checks and the rsync block checksum algorithm entirely and transfer all of the bits at the source to the dest without any rsync CPU penalty. also for this purpose it looks like -H is also required to preserve hard links which the man page notes: "Note that -a does not preserve hardlink…

It sounds like OP had large hard-link counts, since he was using rsnapshot. It's likely that he simply wouldn't have had the space to copy everything over to the new storage without hard-links.

Re: My experience with using cp to copy 432 million files (39 TB)

#158

Post by unwind »

jschwartzi wrote: post #130
robotresearcher wrote: post #87

Earlier quoted context omitted.

> You can't simply nil the reference and expect GC to come through and clean it for you This is exactly what happens when your process exits and its private memory space is freed in its entirety. Hence the discussion. There's no need to clean up before exit, except if you want to (i) use a leak checker; or (ii) run on a machine with no MMU.

It's funny that ii is even an issue because Linux does not support platforms without MMUs without some serious modifications.

But the program being discussed is (of course) GNU cp. That program is not in any way designed for, limited to, or intended only for, use in Linux.

So what Linux supports and doesn't support in terms of memory management really has no bearing on what GNU cp can or cannot do, in general.

Re: My experience with using cp to copy 432 million files (39 TB)

#159

Post by gpvos »

derefr wrote: post #92
pedrocr wrote: post #61

How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all…

If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined. Plan9's default filesystem is made up of two parts: Fossil, and Venti. - Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values. - Venti is a persistent graph database that holds what would today be called "inode met…

For my understanding: what happens if you open a file, change one byte and close it again? Since the SHAsum of the contents has changed, is the entire file now copied?
Post reply on HN