Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

101–110 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#101

Post by derefr »

pedrocr wrote: post #98
derefr wrote: post #92

Earlier quoted context omitted.

If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined. Plan9's default filesystem is made up of two parts: Fossil, and Venti. - Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values. - Venti is a persistent graph database that holds what would today be called "inode met…

> Honestly, I'm terribly confused why all filesystems haven't been broken into these two easily-separable layers. Is it just inertia? The penalty for doing content addressed filesystems is of course the CPU usage. btrfs probably has most of the benefits without the CPU cost with its copy-on-write semantics. Note that what you describe (and my initial process) is a different semantic than hard-links. What you get is s…

In effect, hard links (of mutable files) are a declaration that certain files have the same "identity." You can't get this with plain Venti-on-Fossil, but it's a problem with Fossil (objects are immutable), not with Venti.

Venti-on-Venti-on-Fossil would work, though, since Venti just creates imaginary files that inherit their IO semantics from their underlying store, and this should apply recursively:

1. create two nodes A and B in Venti[1] that refer to one node C in Venti[2], which refers to object[x] with key x in Fossil.

2. Append to A in Venti[1], causing a write to C in Venti[2], causing a write to object[x] Fossil, creating object[y] with key y.

3. Fossil returns y to Venti[2]; Venti[2] updates C to point to object[y] and returns C to Venti[1]; Venti[1] sees that C is unchanged and does nothing.

Now A and B both effectively point to object[y].

(Note that you don't actually have to have two Venti servers for this! There's nothing stopping you from having Venti nodes that refer to other Venti nodes within the same projected filesystem--but since you're exposing these nodes to the user, your get the "dangers" of symbolic links, where e.g. moving them breaks the things that point to them. For IO operations they have the semantics of hard links, though, instead of needing to be special-cased by filesystem-operating syscalls.)

Re: My experience with using cp to copy 432 million files (39 TB)

#102

Post by beachstartup »

mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

in my opinion, raid 10 is by far the best if you have to support the operations of a production site in a real world environment. just keep spare controllers and drives handy. it's just a matter of swapping out failures.

three cost levers: capacity, speed, and downtime.

basically, at some point you will begin to realize the cheapest cost is the capacity itself, everything else is an order of magnitude more difficult to justify. downtime and slowness are simply unacceptable, whereas buying more stuff is perfectly acceptable and quite feasible solution in an enterprise environment.

Re: My experience with using cp to copy 432 million files (39 TB)

#103

Post by pedrocr »

pflanze wrote: post #100
pedrocr wrote: post #61

How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all…

Let's assume we change your algorithm to use inode+device as key instead of the content hash, to make it behave as it 'should'. > constant RAM Well, definitely not constant space . The only reason for it to use less RAM than the hash table is if it uses disk instead of RAM, and that's what happened in op's post, too, by way of swap. You're trading a hash table implemented in user-space for a hash table or b-tree or s…

>The only reason for it to use less RAM than the hash table is if it uses disk instead of RAM, and that's what happened in op's post, too, by way of swap.

Yeah I mentioned it was just an on-disk implementation of cp's in-memory hash table.

>Now what I'm wondering is whether the OP really had a filesystem with basically every file having a hard link count >1

Almost certainly. It was a backup machine with rsnapshot. Most files are just hardlinks to the same file from the previous backup. For his specific case it would probably be easier to code up a domain-specific solution. He probably knows the hard-linked files are all in a regular form like /basepath/YYYY-MM-DD/path/to/file. So he could just iterate /basepath/*/path/to/file and do the correct copy/hardlinks on the destination.

Re: My experience with using cp to copy 432 million files (39 TB)

#104

Post by lamontcg »

golan wrote: post #89
ntsb wrote: post #67

Earlier quoted context omitted.

Wouldn't rsync of been a better and more reliable choice for this?

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

if you want to copy everything and there's nothing at the target:

rsync --whole-file --ignore-times

that should turn off the metadata checks and the rsync block checksum algorithm entirely and transfer all of the bits at the source to the dest without any rsync CPU penalty.

also for this purpose it looks like -H is also required to preserve hard links which the man page notes:

"Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H."

be mildly interesting to see a speed test between rsync with these options and cp.

there are also utilities out there to "post-process" and re-hardlink everything that is identical, so that a fast copy without preserving hardlinks and then a slow de-duplication step would get you to the same endpoint, but at an obvious expense.

Re: My experience with using cp to copy 432 million files (39 TB)

#105

Post by thaumaturgy »

LeoPanthera wrote: post #47
thaumaturgy wrote: post #19

Earlier quoted context omitted.

You're right to recommend a tarpipe. I've had to copy several very large BackupPC storage pools in the past, and a tarpipe is the most reliable way to do it. (The only downside to BackupPC IMO...) For future reference for other folks, the command would look something like this: cd /old-directory && tar czvflpS - . | tar -C /new-directory -xzvf - Tarpipes are especially neat because they can work well over ssh (make s…

The prompt goes to stderr, the pipe only pipes stdout, so a prompt should not cause excessive bonage, as long as you're there to respond to it. Also, don't use -z locally, or even over a moderately fast network. The compression is not that fast and almost always makes things slower.

Good to know!

Also, re: bonage, I agree that it "shouldn't", but it definitely did. From my sysadmin notes file:

> The tar operation kicks off before ssh engages; having ssh ask for a password seems to intermittently cause problems with the tar headers on the receiving end. (It _shouldn't_, but it seems to.)

Re: My experience with using cp to copy 432 million files (39 TB)

#106

Post by thaumaturgy »

hnha wrote: post #29
thaumaturgy wrote: post #19

Earlier quoted context omitted.

You're right to recommend a tarpipe. I've had to copy several very large BackupPC storage pools in the past, and a tarpipe is the most reliable way to do it. (The only downside to BackupPC IMO...) For future reference for other folks, the command would look something like this: cd /old-directory && tar czvflpS - . | tar -C /new-directory -xzvf - Tarpipes are especially neat because they can work well over ssh (make s…

What's the benefit of using a tarpipe locally?

tarpipes can handle gnarly piles of hardlinked files without sweating. BackupPC is free software for network backups that builds a massively hardlinked file pool; tarpipes are the only thing that can reliably copy BackupPC pools from one location to another.

Re: My experience with using cp to copy 432 million files (39 TB)

#107

Post by icedchai »

gizmo686 wrote: post #65
MichaelGG wrote: post #45

Earlier quoted context omitted.

How would such a system even work? Any abnormal process termination or bugs would mean eventually the system becomes unusable.

The system would slowly leak memory until you reboot.

Yep, I remember programming on such a system (Amiga) in the early 90's! Fun times.

Re: My experience with using cp to copy 432 million files (39 TB)

#108

Post by icedchai »

ajross wrote: post #86
icedchai wrote: post #56

For that many files I probably would've used rsync between local disks. shrug

And hopefully you would have written up a similar essay on the oddball experiences you had with rsync, which is even more stateful than cp and even more likely to have odd interactions when used outside its comfort zone. Ditto for tricks like:(cd $src; tar cf - .) | (cd $dst; tar xf -). Pretty much nothing is going to work in an obvious way in a regime like this. That's sort of the point of the article.

Or maybe not. He mentions rsnapshot in the article, which uses rsync under the hood. This implies rsync would have a very good chance of handling a large number of hardlinks... since it created them in the first place.

Re: My experience with using cp to copy 432 million files (39 TB)

#109

Post by hobs »

beachstartup wrote: post #102
mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

in my opinion, raid 10 is by far the best if you have to support the operations of a production site in a real world environment. just keep spare controllers and drives handy. it's just a matter of swapping out failures. three cost levers: capacity, speed, and downtime. basically, at some point you will begin to realize the cheapest cost is the capacity itself, everything else is an order of magnitude more difficult…

Exactly.

I work with a number of medium sized businesses of which you can clearly tell who have had and who havent had major downtime due to these types of hardware failures.

Many people who have never experienced issues have no interest in even forming a basic DR plan, but the ones who have just start throwing money at you the moment you mention it.

Buying more stuff is WAY cheaper than slowness and downtime, and penny pinching on this count is going to cause so much additional heartache for what will in the end cost more anyway.

Re: My experience with using cp to copy 432 million files (39 TB)

#110

Post by breadbox »

3rd3 wrote: post #35
vhost- wrote: post #20

These are the types of stories I love. I just learned a boat load in 5 minutes.

Is there maybe an archive website dedicated to these kind of stories?

At one time there was; it was called the Internet. The archive still exists, but it's been made harder to browse through due to being jumbled up with javascript and cat gifs.
Post reply on HN