Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

201–210 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#201

Post by icedchai »

sophacles wrote: post #198
icedchai wrote: post #108

Earlier quoted context omitted.

Or maybe not. He mentions rsnapshot in the article, which uses rsync under the hood. This implies rsync would have a very good chance of handling a large number of hardlinks... since it created them in the first place.

That doesn't follow. If backups are for multiple machines to a big file server, the backup machine will have a much larger set of files than those that come from an individual machine. Further, each backup "image" compares the directory for the previous backup to the current live system. Generally it looks something like this: 1. Initial backup or "full backup" - copy the full targeted filesystem to the time indexed…

Yes, it has more links, I realize, but this still doesn't mean it wouldn't work. Give it a shot and report back. (Hah.)

Re: My experience with using cp to copy 432 million files (39 TB)

#202

Post by peterwwillis »

asveikau wrote: post #185
pedrocr wrote: post #27

Unix could really use a way to get all the paths that point to a given inode. These days that shouldn't really cost all that much and this issue comes up a lot in copying/sync situations. Here's the git-annex bug report about this: https://git-annex.branchable.com/bugs/Hard_links_not_synced_...

Wow, it's not every day I hear about a filesystem feature that Windows has and Linux doesn't. (On a recent windows system: fsutil hardlink list -- you can try any random exe or DLL in system32 for an example of a hard link.) I forget what the api for that looks like if I ever knew. Might be private. I am surprised, usually Linux is way ahead of Windows on shiny filesystem stuff.

Linux just has more filesystems, and sadly a lot of them have various flaws. I'm surprised when people are surprised that Linux isn't some completely superior technical marvel. BSD and Unix systems have been more advanced for decades..

Everyone on Linux still uses tar for god's sake, even though zip can use the same compression algorithms people use on tarballs, and zip actually stores an index of its files rather than 'cat'ing each record on top of the next like an append-only tape archive. (Obviously there are better formats than 'zip' for any platform, but it's just strange that nobody has moved away from tar)

Re: My experience with using cp to copy 432 million files (39 TB)

#204

Post by brazzy »

grondilu wrote: post #190
brazzy wrote: post #189

Earlier quoted context omitted.

As discussed in detail elswhere abive: the problem is not with the ability to preserve hardlinks, but the resources , namely RAM, required to do so. Two tar command would in fact be much worse, because they'd require twice as many resources.

I doubt it, since you're using a pipe. It's not like you're actually storing the whole archive anywhere. Also, and just to be clear, I did not invent this idea of using tar for this. I remember having read it in a Unix manual when I was learning about this OS. And even if using tar is not a good idea, it should certainly be considered and as such I don't understand why OP doesn't even mention it in his post.

The pipe is not the issue; you actually are storing the full paths of all files in the archive somewhere (namely in RAM) so that you're able to decide whether a given file you're looking at is an alternative hardlink for a file you have already sent.

Yes, using tar for copying is not new and has its use cases, but this is not one of them.

Re: My experience with using cp to copy 432 million files (39 TB)

#205

Post by asveikau »

ygra wrote: post #191
asveikau wrote: post #185

Earlier quoted context omitted.

Wow, it's not every day I hear about a filesystem feature that Windows has and Linux doesn't. (On a recent windows system: fsutil hardlink list -- you can try any random exe or DLL in system32 for an example of a hard link.) I forget what the api for that looks like if I ever knew. Might be private. I am surprised, usually Linux is way ahead of Windows on shiny filesystem stuff.

Is that really a feature the file system provides? Or does that command simply walk the MFT and finds all hardlinks to the same file record? As far as I know NTFS does not store what hard links point to a file.

I don't know. I do know there are tens of thousands of hardlinked files under system32, and the command I listed above can enumerate links pretty quickly.

Re: My experience with using cp to copy 432 million files (39 TB)

#206

Post by ape4 »

rwg wrote: post #122

Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. A wonderful trend I've noticed in Free/Open Source software lately is proudly claiming that a program is "Valgrind clean." It's a decent indication that the program won't doing anything silly with memory during normal use, like leak it. (There's also a notable upswing in the number of projects usi…

Yes, we need a standard arena call(s). As you said, glibc already uses them internally. From the malloc(7) man page:

In a multithreaded application in which threads simultaneously allocate and free memory, there could be contention for these mutexes. To scalably handle memory allocation in multithreaded applications, glibc creates additional memory allocation arenas if mutex contention is detected. Each arena is a large region of memory that is internally allocated by the system (using brk(2) or mmap(2)), and managed with its own mutexes.

Re: My experience with using cp to copy 432 million files (39 TB)

#207

Post by calvins »

I would usually use the tarpipe mentioned already by others for this sort of thing (although I probably wouldn't do 432 million files in one shot):

  (cd $SOURCE && tar cf - .) | (mkdir -p $DEST && cd $DEST && tar xf -)
Another option which I just learned about through reading some links from this thread is pax (http://en.wikipedia.org/wiki/Pax_%28Unix%29), which can do it with just a single process:

  (mkdir -p $DEST && cd $SOURCE && pax -rw . $DEST)
Both will handle hard links fine, but pax may have some advantages in terms of resource usage when processing huge numbers of files and tons of hard links.

Re: My experience with using cp to copy 432 million files (39 TB)

#208

Post by defective »

Forlien wrote: post #139
halfcat wrote: post #137

Earlier quoted context omitted.

This can not be stressed strongly enough. There is never a case when RAID5 is the best choice, ever [1]. There are cases where RAID0 is mathematically proven more reliable than RAID5 [2]. RAID5 should never be used for anything where you value keeping your data. I am not exaggerating when I say that very often, your data is safer on a single hard drive than it is on a RAID5 array. Please let that sink in. The problem…

I think your calculation on failing an array rebuild is wrong. Can you show how you got those numbers?

From this article: http://www.smbitjournal.com/2012/05/when-no-redundancy-is-mo...

Re: My experience with using cp to copy 432 million files (39 TB)

#209

Post by rbh42 »

andreasvc wrote: post #181
steveh73 wrote: post #78

Earlier quoted context omitted.

"Normally I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained the bad blocks."

Given the issues encountered it would make sense to split this into two passes: 1. identify files with bad blocks and log; 2. do the block-level copy. The disadvantages of file-level copy weigh heavier than the doing this in a single step.

That's probably what I'll do if I run into a problem like that another time (I'm the OP).

Re: My experience with using cp to copy 432 million files (39 TB)

#210

Post by rbh42 »

dspillett wrote: post #173

> The number of hard drives flashing red is not the same as the number of hard drives with bad blocks. This is the real take-away. Monitor your drives. At very least enable SMART, and also regularly run a read on the full underlying drive (SMART won't see and log blocks that are on the way out so need retries for successful reads, unless you actually try to read those blocks). That won't completely make you safe, but…

Glad someone noticed it (I'm the OP). Reading the drives systematically is called "Patrol Read" and is often enabled by default, but you can tweak the parameters.
Post reply on HN