Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

141–150 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#141

Post by wtallis »

otakucode wrote: post #126
craigyk wrote: post #83

Earlier quoted context omitted.

My largest ZFS pool is currently ~64TB ( 3 X 10 3TB (raidz2) ) The pool has ranged from 85%-95% full (it's mostly at 85% now and used mostly for reads). Resilvering one drive usually takes Something else cool: When I was planning this out I wrote a little script to calculate the chance of failure. With a drive APF of 10% (which is pretty high), and a rebuild speed of 50MB/sec (very low compared to what I typically ge…

What is APF? And why not use the typical URRE rate to calculate your stats? I'm always mystified at how stupid our storage systems are. Even very expensive SAN solutions from EMC and the like area just... stupid. We've got loads of metrics on every drive, but figuring out that those things should be aggregated and subjected to statistical analysis just seems to have not been done yet. What I really want is a "pasture…

btrfs at least offers the flexibility to use a collection of mismatched drives and ensure that N copies of the data exist across separate devices. Once it's parity-based RAID modes are stable, it should be able to retain that flexibility while offering a redundant option for cases where N=2 wastes too much space. That combined with regular scrubbing should suffice for moderately sized arrays.

Treating drives differently based on their expected failure rates seems like it would only matter for very large arrays trying to keep the redundancy as low as possible.

Re: My experience with using cp to copy 432 million files (39 TB)

#142

Post by Too »

3rd3 wrote: post #35
vhost- wrote: post #20

These are the types of stories I love. I just learned a boat load in 5 minutes.

Is there maybe an archive website dedicated to these kind of stories?

Raymond Chen's The Old New Thing comes to mind, http://blogs.msdn.com/b/oldnewthing/. Mostly windows though.

Re: My experience with using cp to copy 432 million files (39 TB)

#144

Post by beagle3 »

golan wrote: post #89
ntsb wrote: post #67

Earlier quoted context omitted.

Wouldn't rsync of been a better and more reliable choice for this?

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

> In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical.

I regularly rsync machines with millions of files. On a slow portable 2.5" 25MB/sec USB2 connection it's never taken me more than 1hr on completely cold caches to verify that no file needs to be copied. With caches being hot, it's a few minutes. And on faster drives it's faster still.

Unless you are doing something weird and force it to checksum, checksumming mostly kicks in on files that have actually changed and can be transferred efficiently in parts (i.e. network; NOT on local disks). In other cases, it's just a straight copy.

Have you actually used rsync in the setting you describe, or at least read its manual?

Re: My experience with using cp to copy 432 million files (39 TB)

#145

Post by Oletros »

lucb1e wrote: post #16

> 20 years experience with various Unix variants > I browsed the net for other peoples' experience with copying many files and quickly decided that cp would do the job nicely. After 20 years you no longer google how to copy files. Edit: Reading on he talks about strace and even reading cp's source code which makes it even weirder that he had to google how to do this... Edit2: Comments! Took only ten downvotes before…

>> I browsed the net for other peoples' experience with copying many files and quickly decided that cp would do the job nicely

> After 20 years you no longer google how to copy files.

He didn't "google" how to copy files

Re: My experience with using cp to copy 432 million files (39 TB)

#146

Post by khc »

derefr wrote: post #92
pedrocr wrote: post #61

How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all…

If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined. Plan9's default filesystem is made up of two parts: Fossil, and Venti. - Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values. - Venti is a persistent graph database that holds what would today be called "inode met…

Content addressable systems trade CPU and memory with disk space. If you expect duplications to be low, you are usually better off with a background scrubber.

Re: My experience with using cp to copy 432 million files (39 TB)

#147

Post by dmckeon »

grondilu wrote: post #143

On Unix, isn't it considered bad practice to use cp in order to copy a large directory tree? IIRC, the use of tar is recommended. Something like: $ (cd $origin && tar cf - *) | (cd $destination && tar xvf - )

Use && there, not ; - consider the result if either of the cd commands fails.

Re: My experience with using cp to copy 432 million files (39 TB)

#149

Post by fsniper »

minopret wrote: post #132

In light of experience would it perhaps be helpful after all to use a block-level copy (such as Partclone, PartImage, or GNU ddrescue) and analyze later which files have the bad blocks? I see that the choice of a file-level copy was deliberate: "I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained t…

Also there is no mention of unrecoverable file analysis like error handling of cp operations in the article. And with this many files it would not be feasible without using an error log file.

So going with a simple block copy should suffice IMHO.

Re: My experience with using cp to copy 432 million files (39 TB)

#150

Post by khc »

caf wrote: post #131
rwg wrote: post #122

Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. A wonderful trend I've noticed in Free/Open Source software lately is proudly claiming that a program is "Valgrind clean." It's a decent indication that the program won't doing anything silly with memory during normal use, like leak it. (There's also a notable upswing in the number of projects usi…

The "#ifdef DEBUG..." option is pretty much exactly what they did: http://lists.gnu.org/archive/html/coreutils/2014-08/txtVTwv5...

The proper thing to do is include valgrind.h and use the macros to determine if it's running under valgrind. Alternatively to accommodate other tools, hide it behind an env var. We should make it as easy to test production binaries as possible.
Post reply on HN