I wonder how well rsync would have fared here.
Rsync can die just from scanning the whole directory tree of files first.
My experience with using cp to copy 432 million files (39 TB)
21–30 of 267 posts
Re: My experience with using cp to copy 432 million files (39 TB)
#22> 20 years experience with various Unix variants > I browsed the net for other peoples' experience with copying many files and quickly decided that cp would do the job nicely. After 20 years you no longer google how to copy files. Edit: Reading on he talks about strace and even reading cp's source code which makes it even weirder that he had to google how to do this... Edit2: Comments! Took only ten downvotes before…
Because the man is wise. He also didn't kill a job that appeared to be hung, he started reading the code to figure out why and determined that it would in fact, complete.
Re: My experience with using cp to copy 432 million files (39 TB)
#23I would probably have used tar|tar for this, or rsync.
I'm unfamiliar with tar|tar, can you provide some incite into this trick?
Re: My experience with using cp to copy 432 million files (39 TB)
#24> 20 years experience with various Unix variants > I browsed the net for other peoples' experience with copying many files and quickly decided that cp would do the job nicely. After 20 years you no longer google how to copy files. Edit: Reading on he talks about strace and even reading cp's source code which makes it even weirder that he had to google how to do this... Edit2: Comments! Took only ten downvotes before…
I'm not yet 20 years into this game, but the further I go, the more I'm finding I need to be willing to swallow my pride and check in with the current state of best practices.
Re: My experience with using cp to copy 432 million files (39 TB)
#25This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild process if the RAID controller is smart enough. You just swap failed drives, and the RAID controller automatically utilizes the back up drive and then mirrors onto the new drive. Just much faster and much less prone to multiple drive failures killing the entire RAID.
Re: My experience with using cp to copy 432 million files (39 TB)
#26> Wanting the buffers to be flushed so that I had a complete
logfile, I gave cp more than a day to finish disassembling its hash table,
before giving up and killing the process....Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits.
Does anyone know what the 'tear down' part is about? If it's about erasing the hashtable from memory, what takes so long? I would expect that to be very fast: you don't have to write zeros to it all, you just tell your GC or memory manager to mark it as free.
Re: My experience with using cp to copy 432 million files (39 TB)
#27Unix could really use a way to get all the paths that point to a given inode. These days that shouldn't really cost all that much and this issue comes up a lot in copying/sync situations. Here's the git-annex bug report about this:
https://git-annex.branchable.com/bugs/Hard_links_not_synced_...
Re: My experience with using cp to copy 432 million files (39 TB)
#28> Wanting the buffers to be flushed so that I had a complete logfile, I gave cp more than a day to finish disassembling its hash table, before giving up and killing the process....Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. Does anyone know what the 'tear down' part is about? If it's about erasing the hashtable from memory, what takes so lo…
I don't understand it, but the OP says this, implying the author agrees with you for any modern system, but not on 'old systems without working memory management'
> And unless old systems without working memory management must be supported, I don't see any harm in simply removing the call to the forget_all function towards the end of cp.c.
Re: My experience with using cp to copy 432 million files (39 TB)
#29I would probably have used tar|tar for this, or rsync.
You're right to recommend a tarpipe. I've had to copy several very large BackupPC storage pools in the past, and a tarpipe is the most reliable way to do it. (The only downside to BackupPC IMO...) For future reference for other folks, the command would look something like this: cd /old-directory && tar czvflpS - . | tar -C /new-directory -xzvf - Tarpipes are especially neat because they can work well over ssh (make s…
What's the benefit of using a tarpipe locally?
Re: My experience with using cp to copy 432 million files (39 TB)
#30>We use XFS Why?
Most likely because it would take a looong time to fsck 40TB using ext3/4.