Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

81–90 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#81

Post by greenknight »

mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

I had a RAID 6 have a failed disk a couple weeks ago... 8 x 3TB drives... 18TB usable. Took 7 hours to rebuild.

Disks were connected via SATA 3 and was using mdadm

Re: My experience with using cp to copy 432 million files (39 TB)

#82

Post by xenadu02 »

gizmo686 wrote: post #68
noinsight wrote: post #42

Earlier quoted context omitted.

> Teracopy I've used FastCopy for GUI based larger transfers, it's open source and can handle larger datasets well in my experience. It also doesn't choke on >MAX_PATH paths. Haven't had problems with it. Supposedly it's the fastest tool around... The only slight issue is that the author is Japanese so the English translations aren't perfect plus the comments in the source are in Japanese.

">MAX_PATH paths" How does this happen?

Technical debt that keeps on giving.

Today there are N applications. "We can't increase MAX_PATH because it will break existing applications!"

Tomorrow there are N+M applications. "We can't increase MAX_PATH because it will break existing applications!"

Repeat forever.

Any time you are faced with a hard technical decision like this, the pain will always be least if you make the change either:

1. During another transition (e.g. 16-bit to 32-bit, or 32-bit to 64-bit). Microsoft could have required all 64-bit Windows apps to adopt a larger MAX_PATH, among other things.

2. Right NOW, because there will never be an easier time to make the change. The overall pain to all parties will only increase over time.

Re: My experience with using cp to copy 432 million files (39 TB)

#83

Post by craigyk »

LeoPanthera wrote: post #40
mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

How does the rebuild time of ZFS's raidz compare to RAID5/6?

My largest ZFS pool is currently ~64TB ( 3 X 10 3TB (raidz2) ) The pool has ranged from 85%-95% full (it's mostly at 85% now and used mostly for reads).

Resilvering one drive usually takes Something else cool: When I was planning this out I wrote a little script to calculate the chance of failure. With a drive APF of 10% (which is pretty high), and a rebuild speed of 50MB/sec (very low compared to what I typically get) I have a 1% chance of losing the entire pool over a 46.6 year period. If I add 4 more raidz2 10X3TB VDEVS that would drop to 3.75 years.

Re: My experience with using cp to copy 432 million files (39 TB)

#84

Post by drv »

gizmo686 wrote: post #68
noinsight wrote: post #42

Earlier quoted context omitted.

> Teracopy I've used FastCopy for GUI based larger transfers, it's open source and can handle larger datasets well in my experience. It also doesn't choke on >MAX_PATH paths. Haven't had problems with it. Supposedly it's the fastest tool around... The only slight issue is that the author is Japanese so the English translations aren't perfect plus the comments in the source are in Japanese.

">MAX_PATH paths" How does this happen?

This article has a pretty nice overview: http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

The MAX_PATH limitation can be avoided by using the \\?\ prefix (which also disables some kinds of path normalization, like converting / to \) or by using paths relative to the current directory.

Unfortunately, most tools (including Windows Explorer and other Microsoft software) still don't work with paths longer than MAX_PATH.

Re: My experience with using cp to copy 432 million files (39 TB)

#85

Post by mjn »

andreasvc wrote: post #72
mjn wrote: post #34

Earlier quoted context omitted.

Looking at the code, it looks like deallocating a hash table requires traversing the entire table, because there is malloc()'d memory associated with each hash entry, so each entry has to be visited and free()'d. From hash_free() in coreutils hash.c: for (bucket = table->bucket; bucket bucket_limit; bucket++) { for (cursor = bucket->next; cursor; cursor = next) { next = cursor->next; free (cursor); } } Whereas if you…

Isn't it a terribly inefficient design to malloc hash entries individually? It would make more sense to use a pool / arena for that.

You could, but it's fairly uncommon to do that kind of "custom allocator" style in C, especially in traditional Unix utility programming. It seems to be more of a C++ thing culturally in my experience, though you do find it in C in "big" programs, like scientific-computing code.

Re: My experience with using cp to copy 432 million files (39 TB)

#86

Post by ajross »

icedchai wrote: post #56

For that many files I probably would've used rsync between local disks. shrug

And hopefully you would have written up a similar essay on the oddball experiences you had with rsync, which is even more stateful than cp and even more likely to have odd interactions when used outside its comfort zone.

Ditto for tricks like:(cd $src; tar cf - .) | (cd $dst; tar xf -).

Pretty much nothing is going to work in an obvious way in a regime like this. That's sort of the point of the article.

Re: My experience with using cp to copy 432 million files (39 TB)

#87

Post by robotresearcher »

lgg wrote: post #37
gwern wrote: post #26

> Wanting the buffers to be flushed so that I had a complete logfile, I gave cp more than a day to finish disassembling its hash table, before giving up and killing the process....Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. Does anyone know what the 'tear down' part is about? If it's about erasing the hashtable from memory, what takes so lo…

This is system C the memory is manually managed with malloc(). You can't simply nil the reference and expect GC to come through and clean it for you, you have to issue a free() for each malloc() that was called. While it might be possible to write a custom disposal routine that does that efficiently for the structure, the simpler way is to just use the normal functions you use for removing single items and call it ag…

> You can't simply nil the reference and expect GC to come through and clean it for you

This is exactly what happens when your process exits and its private memory space is freed in its entirety. Hence the discussion. There's no need to clean up before exit, except if you want to (i) use a leak checker; or (ii) run on a machine with no MMU.

Re: My experience with using cp to copy 432 million files (39 TB)

#88

Post by fspeech »

jrochkind1 wrote: post #41

So it was all the files in one go, presumably with `cp -r`? What about doing something with find/xargs/i-dunno to copy all the files, but break em into batches so you aren't asking cp to do it's bookkeeping for so many files in one process? Would that work better? Or worse in other ways?

How do you keep hard links?

Re: My experience with using cp to copy 432 million files (39 TB)

#89

Post by golan »

ntsb wrote: post #67
mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

Wouldn't rsync of been a better and more reliable choice for this?

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical.

Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times.

Add checksumming, and you probably would like to take a holiday whilst it copies the data :-)

Re: My experience with using cp to copy 432 million files (39 TB)

#90

Post by sounds »

xchg_ax_ax wrote: post #77
jrochkind1 wrote: post #41

So it was all the files in one go, presumably with `cp -r`? What about doing something with find/xargs/i-dunno to copy all the files, but break em into batches so you aren't asking cp to do it's bookkeeping for so many files in one process? Would that work better? Or worse in other ways?

This page may be useful: http://unix.stackexchange.com/questions/44247/how-to-copy-di... The main issue is that there's no api to get the list of files hard linked together: the only way is to check all the existing files and compare inodes. If you're doing a plain copy over 2 fs, you cannot choose which number the target inode will be, so you need to keep a map between inode numbers, or between inodes and file names…

pedrocr's comment above suggests a good solution:

1. Copy each file from the source volume to a single directory (e.g. /tmp) on the target volume, named for the source volume inode number.

(edit: I suggest using a hierarchy of dirs to avoid the "too many dentry's" slowdown)

2. If the file has already been copied, it will already exist in /tmp - looking up the inode is a vanilla directory lookup

3. Create a hard link from /tmp to the actual path of the file

4. When all the files have been created on the target volume, delete the inode numbers in /tmp

Post reply on HN