Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

111–120 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#111

Post by t0mas88 »

mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

There is even a community for it: BAARF - http://www.miracleas.com/BAARF/

The main reason is not only the rebuild time (which indeed is horrible for a loaded system), but also the performance characteristics that can be terrible if you do a lot of writing.

We for example more than doubled the throughput of Cassandra by dropping RAID5. So the saving of 2 disks per server in the end required buying almost twice as many servers.

Re: My experience with using cp to copy 432 million files (39 TB)

#112

Post by jayvanguard »

breadbox wrote: post #110
3rd3 wrote: post #35

Earlier quoted context omitted.

Is there maybe an archive website dedicated to these kind of stories?

At one time there was; it was called the Internet. The archive still exists, but it's been made harder to browse through due to being jumbled up with javascript and cat gifs.

It's true. We should have never let the public on the Internet. It has been downhill since then.

Re: My experience with using cp to copy 432 million files (39 TB)

#114

Post by mrb »

pflanze wrote: post #100
pedrocr wrote: post #61

How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all…

Let's assume we change your algorithm to use inode+device as key instead of the content hash, to make it behave as it 'should'. > constant RAM Well, definitely not constant space . The only reason for it to use less RAM than the hash table is if it uses disk instead of RAM, and that's what happened in op's post, too, by way of swap. You're trading a hash table implemented in user-space for a hash table or b-tree or s…

In fact, 17e9/432e6=39.35 B/file, pretty spot on. Seems like there's some room for optimization

It does not suggest there is a possible optimization. cp needs to store the pointer and the path itself in memory, so it is more than 8 bytes. ~39 bytes/file means the average path length was 15 bytes (8-byte source device + 8-byte source inode + 8-byte pointer + ~15-byte path = ~39 bytes). I guess one theoretical optimization would be to add to Linux a system call to open a file by device+inode number. So the pointer+path would be replaced by a fixed 8-byte dest device + 8-byte dest inode (so 32 bytes/file). Or since the destination is only one filesystem in this case, you would just need 8-byte source device + 8-byte source inode + 8-byte dest inode (24 bytes/file). In this latter case, the 17 GB hash table would have shrunk to 10 GB, which could have help completely avoid the swap in the OP's case.

Edit: correct, I do not account for unused space in the hash table, for simplicity. So read all my numbers assuming a ±10-30% range. However it seems that the OP who quoted 432e6 files meant 432e6 file names, not inodes. So if the average inode had, say, 4.3 hardlinks (IOW 4.3 names), it means the 17GB hashtable only had 100e6 files in it, so 170 bytes/file, so 170-8-8-8 = 146-byte pathnames on average (±10-30%), with sounds more reasonable.

Re: My experience with using cp to copy 432 million files (39 TB)

#115

Post by laymil »

fintler wrote: post #93

I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/h…

And it's interesting and useful for scientific computing where you already have an MPI environment and distributed/parallel filesystems. However, it's not really applicable to this workload, as the paper itself says.

There is a provision in most file systems to use links (symlinks, hardlinks, etc.). Links can cause cycles in the file tree, which would result in a traversal algorithm going into an infinite loop. To prevent this from happening, we ignore links in the file tree during traversal. We note that the algorithms we propose in the paper will duplicate effort proportional to the number of hardlinks. However, in real world production systems, such as in LANL (and others), for simplicity, the parallel filesystems are generally not POSIX compliant, that is, they do not use hard links, inodes, and symlinks. So, our assumption holds.

The reason this cp took such large amounts of time was the desire to preserve hardlinks and the resize of the hashtable used to track the device and inode of the source and destination files.

Re: My experience with using cp to copy 432 million files (39 TB)

#116

Post by mililani »

click170 wrote: post #99
greenknight wrote: post #81

Earlier quoted context omitted.

I had a RAID 6 have a failed disk a couple weeks ago... 8 x 3TB drives... 18TB usable. Took 7 hours to rebuild. Disks were connected via SATA 3 and was using mdadm

In my experience working with Debian, the rebuild is sensitive to disk usage. If you start using the disks, the rebuild will slow down intentionally so as to not choke the drives. This could explain why it was fast for you but not for the parent. Edit: To clarify, I was using mdadm, not hardware raid.

Yeah, I question that rebuild. If the array is being used along with the rebuild, typically the rebuild process gets put to a lower priority. I've seen IBM enterprise RAID 5 storage arrays take forever to rebuild if they were the main storage for a high transaction database.

Re: My experience with using cp to copy 432 million files (39 TB)

#117

Post by bakul »

derefr wrote: post #92
pedrocr wrote: post #61

How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all…

If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined. Plan9's default filesystem is made up of two parts: Fossil, and Venti. - Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values. - Venti is a persistent graph database that holds what would today be called "inode met…

1. Plan9 has no hard links so if you copy unix dir. tree to a plan9 machine you'd lose all the hard link info. 2. Venti doesn't use fossil or inodes. Venti is just content addressable storage system; not a fileserver/system. 3. Fossil is a fileserver.

Re: My experience with using cp to copy 432 million files (39 TB)

#118

Post by im3w1l »

noinsight wrote: post #42
IvyMike wrote: post #10

Interesting. In Windows-land, the default copy is pretty anemic, so probably most people avoid it for serious work. I'd probably use robocopy from the command line. And if I was being lazy, I'd use the Teracopy GUI. I think my limit for a single copy command has been around 4TB with robocopy--and that was a bunch of large media files, instead of smaller more numerous files. Maybe there's a limit I haven't hit.

> Teracopy I've used FastCopy for GUI based larger transfers, it's open source and can handle larger datasets well in my experience. It also doesn't choke on >MAX_PATH paths. Haven't had problems with it. Supposedly it's the fastest tool around... The only slight issue is that the author is Japanese so the English translations aren't perfect plus the comments in the source are in Japanese.

But, can it handle the real stress test: filenames with colons?

Re: My experience with using cp to copy 432 million files (39 TB)

#119

Post by thrownaway2424 »

thrownaway2424 wrote: post #12
gaius wrote: post #2

I would probably have used tar|tar for this, or rsync.

I would certainly have used cpio.

Down voted because you disagree? Or you don't know what cpio is? Either way please discuss instead of or in addition to voting.

Re: My experience with using cp to copy 432 million files (39 TB)

#120

Post by lunixbochs »

lamontcg wrote: post #104
golan wrote: post #89

Earlier quoted context omitted.

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

if you want to copy everything and there's nothing at the target: rsync --whole-file --ignore-times that should turn off the metadata checks and the rsync block checksum algorithm entirely and transfer all of the bits at the source to the dest without any rsync CPU penalty. also for this purpose it looks like -H is also required to preserve hard links which the man page notes: "Note that -a does not preserve hardlink…

I use rsync to replicate 10TB+ volumes without any problems. It's also very fast to catch up on repeat runs, so you can ^C it without being too paranoid.
Post reply on HN