Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

171–180 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#171

Post by dspillett »

greenknight wrote: post #81
mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

I had a RAID 6 have a failed disk a couple weeks ago... 8 x 3TB drives... 18TB usable. Took 7 hours to rebuild. Disks were connected via SATA 3 and was using mdadm

Assuming it is reading and writing almost simultaneously that is ~125Mbyte/sec on each physical drive, which isn't bad considering the latency of the read-from-three-or-four-then-checksum-then-write-to-the-other-one-or-two process if you if the array was was in use at the time.

There are a few ways to improve RAID rebuild time (see http://www.cyberciti.biz/tips/linux-raid-increase-resync-reb... for example) but be aware that they tend to negatively affect performance of other use of the array while the build is happening (that is, you are essentially telling the kernel to give more priority to the rebuild process than it normally would when there are other processes accessing the data too).

Re: My experience with using cp to copy 432 million files (39 TB)

#173

Post by dspillett »

> The number of hard drives flashing red is not the same as the number of hard drives with bad blocks.

This is the real take-away. Monitor your drives. At very least enable SMART, and also regularly run a read on the full underlying drive (SMART won't see and log blocks that are on the way out so need retries for successful reads, unless you actually try to read those blocks).

That won't completely make you safe, but it'll greatly reduce the risk of other drives failing during a rebuild by increasing the chance you get advanced warning that problems are building up.

Re: My experience with using cp to copy 432 million files (39 TB)

#174

Post by andreasvc »

_delirium wrote: post #128
rwg wrote: post #122

Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. A wonderful trend I've noticed in Free/Open Source software lately is proudly claiming that a program is "Valgrind clean." It's a decent indication that the program won't doing anything silly with memory during normal use, like leak it. (There's also a notable upswing in the number of projects usi…

Adding some kind of arena-allocation library to both the build & runtime dependencies solely to keep valgrind happy, with no actual improvement in functionality or performance, doesn't seem like a great tradeoff on the software engineering front. I'd rather see work on improving the static analysis. For example if some memory is intended to be freed at program cleanup, Valgrind could have some way of being told, "thi…

I don't see why you assume arenas would be added "solely to keep valgrind happy". Arenas have better performance when allocating a high number of small chunks, because an arena can make better performance trade-offs for this use case than the general malloc allocator.

Re: My experience with using cp to copy 432 million files (39 TB)

#175

Post by ccozan »

brazzy wrote: post #168
grondilu wrote: post #143

On Unix, isn't it considered bad practice to use cp in order to copy a large directory tree? IIRC, the use of tar is recommended. Something like: $ (cd $origin && tar cf - *) | (cd $destination && tar xvf - )

Would have been even worse in this case due to the requirement to preserve hard links.

tar can preserve hardlinks. In fact, tar is really the best solution in this case, using the streaming between two tar commands.

Re: My experience with using cp to copy 432 million files (39 TB)

#176

Post by bhaak »

icedchai wrote: post #108
ajross wrote: post #86

Earlier quoted context omitted.

And hopefully you would have written up a similar essay on the oddball experiences you had with rsync, which is even more stateful than cp and even more likely to have odd interactions when used outside its comfort zone. Ditto for tricks like:(cd $src; tar cf - .) | (cd $dst; tar xf -). Pretty much nothing is going to work in an obvious way in a regime like this. That's sort of the point of the article.

Or maybe not. He mentions rsnapshot in the article, which uses rsync under the hood. This implies rsync would have a very good chance of handling a large number of hardlinks... since it created them in the first place.

But rsnapshot created that large number of hardlinks incrementally. Without trying it there's no indication that rsync handles that better, especially as rsync with default settings doesn't preserve hardlinks as rsync builds a quite expensive data structure for them.

I'm surprised that cp even worked. I wouldn't have had the balls to just use cp as it could just have failed.

My strategy would probably have been to cp or rsync over the latest backup (so that new incremental backups could be immediately be created again) and then incrementally rsync'ing the previous backups with hardlink detection.

Re: My experience with using cp to copy 432 million files (39 TB)

#177

Post by tfinniga »

rwg wrote: post #122

Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. A wonderful trend I've noticed in Free/Open Source software lately is proudly claiming that a program is "Valgrind clean." It's a decent indication that the program won't doing anything silly with memory during normal use, like leak it. (There's also a notable upswing in the number of projects usi…

The advantage of the #ifdef DEBUG method is that it's possible to ensure that all of your allocations are clean at the object level, not just at the arena level.

In the limit, you could make one arena for your entire program and then deallocate it on process exit (essentially what the OS gives you by default). Using smaller and smaller arenas lets you see that at a finer and finer granularity. In the other limit you're back to the per-object allocations.

Using an arena to avoid per-object leak checking is good if you're absolutely sure that there are no leaks in that block of code. You're essentially hiding any leaks within the arena from valgrind.

But that seems dangerous, because it makes it harder to use valgrind for what it's good at: finding leaks that you're unaware of, especially in code that you 'know' is good.

Re: My experience with using cp to copy 432 million files (39 TB)

#178

Post by pedrocr »

perlgeek wrote: post #151
pedrocr wrote: post #61

How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all…

This has the slight flaw that you create two copies instead of one for each file (double write IO, double number of inodes) if there aren't many hardlinks. I don't think that's an acceptable perfomance penalty in most situations.

That won't happen. I'm suggesting using hard-links so you only write the file once to one inode, you just create two files pointing to that inode. You can even optimize that a bit and not even create the two files when the inode has a hard-link count of 1.

Re: My experience with using cp to copy 432 million files (39 TB)

#179

Post by KaiserPro »

golan wrote: post #89
ntsb wrote: post #67

Earlier quoted context omitted.

Wouldn't rsync of been a better and more reliable choice for this?

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

WE use 10 rsyncs in parallel to copy .5pb in less than 3 days.

CPU is cheap.

Post reply on HN