This is not, not, not how one should be using RAID. The math is clear that in sufficiently large disk systems, RAID5, RAID6, and friends, are all insufficient.
Can you elaborate?
My experience with using cp to copy 432 million files (39 TB)
211–220 of 267 posts
Re: My experience with using cp to copy 432 million files (39 TB)
#212In light of experience would it perhaps be helpful after all to use a block-level copy (such as Partclone, PartImage, or GNU ddrescue) and analyze later which files have the bad blocks? I see that the choice of a file-level copy was deliberate: "I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained t…
Also there is no mention of unrecoverable file analysis like error handling of cp operations in the article. And with this many files it would not be feasible without using an error log file. So going with a simple block copy should suffice IMHO.
I'm the OP, so I can shred a bit of light on that: Dell's support suggested a file-level copy when I asked them what they recommended (but I'm not entirely sure they understood the implications). Also, time was not a big issue.
I did keep a log file with the output from cp, and it clearly identified the filenames for the inodes with bad blocks. Actually, I'm not sure how dd would handle bad blocks.
Re: My experience with using cp to copy 432 million files (39 TB)
#213I would usually use the tarpipe mentioned already by others for this sort of thing (although I probably wouldn't do 432 million files in one shot): (cd $SOURCE && tar cf - .) | (mkdir -p $DEST && cd $DEST && tar xf -) Another option which I just learned about through reading some links from this thread is pax ( http://en.wikipedia.org/wiki/Pax_%28Unix%29 ), which can do it with just a single process: (mkdir -p $DEST…
You know how tar handles hardlinks, right? By creating a giant hash table of every file.
Re: My experience with using cp to copy 432 million files (39 TB)
#214Earlier quoted context omitted.
What's the benefit of using a tarpipe locally?
No giant bookkeeping datastructures that end up with the process thrashing in swap, because tar is designed to work with so many files that you don't want to keep the metadata in memory.
How does tar keep track of every hardlink without a giant hash table? It doesn't. It's not magic.
Re: My experience with using cp to copy 432 million files (39 TB)
#215>We use XFS Why?
Tested by time (I'm the OP).
Re: My experience with using cp to copy 432 million files (39 TB)
#216In light of experience would it perhaps be helpful after all to use a block-level copy (such as Partclone, PartImage, or GNU ddrescue) and analyze later which files have the bad blocks? I see that the choice of a file-level copy was deliberate: "I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained t…
A while back I had a hard drive die with an unrecoverable error on the superblock (ext3) and a few other places. I sent the hard drive to a professional recovery firm who were as incompetent as incompetent can be. I got the duff drive back and decided to have a go myself. Recovering the superblock was easy - there are extra copies of that. I then did a full map of the drive listing bad blocks, and did a reverse map to work out which files contained those blocks. Luckily, the only file that contained any bad blocks was a swap file, so I recovered absolutely everything. Other bad blocks were in unallocated space. Ext3 is actually a sane enough disc layout that it is possible to work out what is going on manually. I would have been stuffed if it had been a more complicated filesystem.
Re: My experience with using cp to copy 432 million files (39 TB)
#217This is not, not, not how one should be using RAID. The math is clear that in sufficiently large disk systems, RAID5, RAID6, and friends, are all insufficient.
Can you elaborate?
I'm guessing he means that RAID is designed as a technology to increase the availability of your data and the person in this story is using it as a backup solution. Google 'RAID backup' to see why it's such a bad idea to use it for that purpose.
He specifically used cp because he was worried about data loss from using something like dd. But if he'd been doing regular backups to tape, he could have just restored the backup from tape with no fuss or problems. It's only because he was using tools that aren't suited to what he was trying to do that he ran into problems.
Re: My experience with using cp to copy 432 million files (39 TB)
#218Feels like a similar situation to this: http://dis.4chan.org/read/prog/1109211978/21
Re: My experience with using cp to copy 432 million files (39 TB)
#219Earlier quoted context omitted.
Wow, it's not every day I hear about a filesystem feature that Windows has and Linux doesn't. (On a recent windows system: fsutil hardlink list -- you can try any random exe or DLL in system32 for an example of a hard link.) I forget what the api for that looks like if I ever knew. Might be private. I am surprised, usually Linux is way ahead of Windows on shiny filesystem stuff.
Linux just has more filesystems, and sadly a lot of them have various flaws. I'm surprised when people are surprised that Linux isn't some completely superior technical marvel. BSD and Unix systems have been more advanced for decades.. Everyone on Linux still uses tar for god's sake, even though zip can use the same compression algorithms people use on tarballs, and zip actually stores an index of its files rather th…
tar is good enough for many uses, so people did not move on.
And it doesn't help that tar.gz / tar.bz2 compresses way better than zip in most cases (thanks to using a single compression context, rather than a new one for each file; and also compressing the filenames in the same context), and that it carries ownership and permission information with it - whereas zip doesn't.
The HVSC project, who try to collect every single piece of music ever created on a Commodore C64, distribute their archive as a zip-within-a-zip. The common music file is 1k-4k, goes down to ~500-1000 bytes zipped; The subdirectory+filename are often 100 bytes with a lot of redundancy that zip doesn't use, so they re-zip. Had they used .tar.gz or .tar.bz2, the second stage would not be needed.
Re: My experience with using cp to copy 432 million files (39 TB)
#220I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/h…
And it's interesting and useful for scientific computing where you already have an MPI environment and distributed/parallel filesystems. However, it's not really applicable to this workload, as the paper itself says. There is a provision in most file systems to use links (symlinks, hardlinks, etc.). Links can cause cycles in the file tree, which would result in a traversal algorithm going into an infinite loop. To pr…
Author of the paper here. The file operations are distributed strictly without links, otherwise we could make no guarantees that work wouldn't be duplicated, or even that the algorithm would terminate. We were lucky in that because the parallel file system itself wasn't POSIX, so we didn't have to make our tools POSIX either.