Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

131–140 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#131

Post by caf »

rwg wrote: post #122

Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. A wonderful trend I've noticed in Free/Open Source software lately is proudly claiming that a program is "Valgrind clean." It's a decent indication that the program won't doing anything silly with memory during normal use, like leak it. (There's also a notable upswing in the number of projects usi…

The "#ifdef DEBUG..." option is pretty much exactly what they did:

http://lists.gnu.org/archive/html/coreutils/2014-08/txtVTwv5...

Re: My experience with using cp to copy 432 million files (39 TB)

#132

Post by minopret »

In light of experience would it perhaps be helpful after all to use a block-level copy (such as Partclone, PartImage, or GNU ddrescue) and analyze later which files have the bad blocks?

I see that the choice of a file-level copy was deliberate: "I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained the bad blocks."

Re: My experience with using cp to copy 432 million files (39 TB)

#133

Post by encoderer »

laymil wrote: post #115
fintler wrote: post #93

I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/h…

And it's interesting and useful for scientific computing where you already have an MPI environment and distributed/parallel filesystems. However, it's not really applicable to this workload, as the paper itself says. There is a provision in most file systems to use links (symlinks, hardlinks, etc.). Links can cause cycles in the file tree, which would result in a traversal algorithm going into an infinite loop. To pr…

Sure, but if you read that article you walk away with a sense of thats a lot of files to copy. And the GP built a tool for jobs 2-3 orders of magnitude larger?! Clearly there are tradeoffs forced on you at that size...

Re: My experience with using cp to copy 432 million files (39 TB)

#134

Post by caf »

ccleve wrote: post #129

Maybe this is naive, but wouldn't it have made more sense to do a bunch of smaller cp commands? Like sweep through the directory structure and do one cp per directory? Or find some other way to limit the number of files copied per command?

No, because then it wouldn't have replicated the hardlink structure of the original tree. That was the goal, and also the bit that causes the high resource consumption.

Re: My experience with using cp to copy 432 million files (39 TB)

#135

Post by voltagex_ »

thrownaway2424 wrote: post #119
thrownaway2424 wrote: post #12

Earlier quoted context omitted.

I would certainly have used cpio.

Down voted because you disagree? Or you don't know what cpio is? Either way please discuss instead of or in addition to voting.

Your post has no information as to why you'd use cpio.

Re: My experience with using cp to copy 432 million files (39 TB)

#136

Post by ori_b »

derefr wrote: post #101
pedrocr wrote: post #98

Earlier quoted context omitted.

> Honestly, I'm terribly confused why all filesystems haven't been broken into these two easily-separable layers. Is it just inertia? The penalty for doing content addressed filesystems is of course the CPU usage. btrfs probably has most of the benefits without the CPU cost with its copy-on-write semantics. Note that what you describe (and my initial process) is a different semantic than hard-links. What you get is s…

In effect, hard links (of mutable files) are a declaration that certain files have the same "identity." You can't get this with plain Venti-on-Fossil, but it's a problem with Fossil (objects are immutable), not with Venti. Venti-on-Venti-on-Fossil would work, though, since Venti just creates imaginary files that inherit their IO semantics from their underlying store, and this should apply recursively: 1. create two n…

You seem to be confusing venti and fossil.

Re: My experience with using cp to copy 432 million files (39 TB)

#137

Post by halfcat »

mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

This can not be stressed strongly enough. There is never a case when RAID5 is the best choice, ever [1]. There are cases where RAID0 is mathematically proven more reliable than RAID5 [2]. RAID5 should never be used for anything where you value keeping your data. I am not exaggerating when I say that very often, your data is safer on a single hard drive than it is on a RAID5 array. Please let that sink in.

The problem is that once a drive fails, during the rebuild, if any of the surviving drives experience an unrecoverable read error (URE), the entire array will fail. On consumer-grade SATA drives that have a URE rate of 1 in 10^14, that means if the data on the surviving drives totals 12TB, the probability of the array failing rebuild is close to 100%. Enterprise SAS drives are typically rated 1 URE in 10^15, so you improve your chances ten-fold. Still an avoidable risk.

RAID6 suffers from the same fundamental flaw as RAID5, but the probability of complete array failure is pushed back one level, making RAID6 with enterprise SAS drives possibly acceptable in some cases, for now (until hard drive capacities get larger).

I no longer use parity RAID. Always RAID10 [3]. If a customer insists on RAID5, I tell them they can hire someone else, and I am prepared to walk away.

I haven't even touched on the ridiculous cases where it takes RAID5 arrays weeks or months to rebuild, while an entire company limps inefficiently along. When productivity suffers company-wide, the decision makers wish they had paid the tiny price for a few extra disks to do RAID10.

In the article, he has 12x 4TB drives. Once two drives failed, assuming he is using enterprise drives (Dell calls them "near-line SAS", just an enterprise SATA), there is a 33% chance the entire array fails if he tries to rebuild. If the drives are plain SATA, there is almost no chance the array completes a rebuild.

[1] http://www.smbitjournal.com/2012/11/choosing-a-raid-level-by...

[2] http://www.smbitjournal.com/2012/05/when-no-redundancy-is-mo...

[3] http://www.smbitjournal.com/2012/11/one-big-raid-10-a-new-st...

Re: My experience with using cp to copy 432 million files (39 TB)

#138

Post by thrownaway2424 »

voltagex_ wrote: post #135
thrownaway2424 wrote: post #119

Earlier quoted context omitted.

Down voted because you disagree? Or you don't know what cpio is? Either way please discuss instead of or in addition to voting.

Your post has no information as to why you'd use cpio.

OK, fair point. cpio doesn't have global data structures that grow with the size of the operation, which cp does have. rsync prior to 3.0 also had to scan the entire job before it would begin the transfer, which can take forever. cpio operates on a stream, in the form of find | cpio , so cpio only sees one file from the stream at a time. Find, at least GNU find, is also written such that its resources don't vary with the amount of output.

rsync is a bit of a kitchen sink, and cp is too simple. cpio is just the thing, and was designed for this exact use case.

Re: My experience with using cp to copy 432 million files (39 TB)

#139

Post by Forlien »

halfcat wrote: post #137
mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

This can not be stressed strongly enough. There is never a case when RAID5 is the best choice, ever [1]. There are cases where RAID0 is mathematically proven more reliable than RAID5 [2]. RAID5 should never be used for anything where you value keeping your data. I am not exaggerating when I say that very often, your data is safer on a single hard drive than it is on a RAID5 array. Please let that sink in. The problem…

I think your calculation on failing an array rebuild is wrong. Can you show how you got those numbers?

Re: My experience with using cp to copy 432 million files (39 TB)

#140

Post by squiguy7 »

breadbox wrote: post #110
3rd3 wrote: post #35

Earlier quoted context omitted.

Is there maybe an archive website dedicated to these kind of stories?

At one time there was; it was called the Internet. The archive still exists, but it's been made harder to browse through due to being jumbled up with javascript and cat gifs.

"due to being jumbled with javascript..." - This made my day.
Post reply on HN