Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

231–240 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#231

Post by dredmorbius »

The email states that file-based copy operations were used in favor of dd due to suspected block errors. Two questions come to mind:

1. I've not used dd on failing media, so I'm not sure of the behavior. Will it plow through a file with block-read failures or halt?

2. There's the ddrescue utility, which is specifically intended for reading from nonreliable storage. Seems that this could have offered another means for addressing Rasmus's problem. It can also fill in additional data on multiple runs across media, such that more complete restores might be achieved. https://www.gnu.org/software/ddrescue/ddrescue.html

Re: My experience with using cp to copy 432 million files (39 TB)

#232

Post by pflanze »

dredmorbius wrote: post #231

The email states that file-based copy operations were used in favor of dd due to suspected block errors. Two questions come to mind: 1. I've not used dd on failing media, so I'm not sure of the behavior. Will it plow through a file with block-read failures or halt? 2. There's the ddrescue utility, which is specifically intended for reading from nonreliable storage. Seems that this could have offered another means for…

OP said "I went for a file-level copy because then I'd know which files contained the bad blocks". When you copy the block device with ddrescue (dd doesn't have logic to work around the bad sectors and the only sensible action for it is thus to stop, but don't take my word for it), the result will just have zeroes in the places where bad blocks were, and, assuming the filesystem structure is good enough (you should run fsck on it), will give you files with zones of zeroes. But you won't know which files without either comparing them to a backup (which you won't have by definition if you're trying to recover) or with a program that verifies every file's structure (which won't exist for the general case). Whereas cp will issue error messages with the path of the file in question. So the OP's decision makes sense.

Re: My experience with using cp to copy 432 million files (39 TB)

#233

Post by tedunangst »

dredmorbius wrote: post #229
tedunangst wrote: post #213

Earlier quoted context omitted.

You know how tar handles hardlinks, right? By creating a giant hash table of every file.

How's that going to scale with memory? In-memory hash tables were the downfall of cp here.

It's going to scale just like you'd imagine it would. All the people saying "oh, tar was built for this" obviously haven't actually tried replicating the experiment using tar.

Re: My experience with using cp to copy 432 million files (39 TB)

#234

Post by thinkling »

taeric wrote: post #221
jayvanguard wrote: post #112

Earlier quoted context omitted.

It's true. We should have never let the public on the Internet. It has been downhill since then.

False choice, isn't it? I mean, the complaint isn't that the public now has sites with massive javascript and related technologies. The complaint is that it has muscled out useful sites that did not use those technologies. And it should be heavily noted that the heavy muscles that have pushed out many of these sites is not necessarily "the public."

Kind of funny to say that on a text-only JS-free site that seems to be alive and well, linking to an article on an old-school mailing list archive site. :)

Re: My experience with using cp to copy 432 million files (39 TB)

#235

Post by taeric »

thinkling wrote: post #234
taeric wrote: post #221

Earlier quoted context omitted.

False choice, isn't it? I mean, the complaint isn't that the public now has sites with massive javascript and related technologies. The complaint is that it has muscled out useful sites that did not use those technologies. And it should be heavily noted that the heavy muscles that have pushed out many of these sites is not necessarily "the public."

Kind of funny to say that on a text-only JS-free site that seems to be alive and well, linking to an article on an old-school mailing list archive site. :)

Oh, certainly. I just can resonate with the sentiment that these sites aren't the majority.

Even this site, honestly, is less than easy to deal with on a recurring basis. (Consider, hard to remember which was the top story three days ago at noon.) Specifically, sometimes I lose a story because I refresh and something plummeted off the page. Hard to have any idea how far to "scroll back" to see it.

Re: My experience with using cp to copy 432 million files (39 TB)

#236

Post by DougBTX »

dredmorbius wrote: post #230
fintler wrote: post #93

I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/h…

NB, that PDF seems to have a number of formatting glitches, e.g., the first sentence: "The amount of scienti c data". Numerous others as well, under both xpdf and evince.

There is an fi ligature between the i and the c, perhaps those PDF renderers don't support them? Could be some sort of font loading issue.

Re: My experience with using cp to copy 432 million files (39 TB)

#237

Post by dredmorbius »

DougBTX wrote: post #236
dredmorbius wrote: post #230

Earlier quoted context omitted.

NB, that PDF seems to have a number of formatting glitches, e.g., the first sentence: "The amount of scienti c data". Numerous others as well, under both xpdf and evince.

There is an fi ligature between the i and the c, perhaps those PDF renderers don't support them? Could be some sort of font loading issue.

Perhaps. That's only one of many similar issues.

Re: My experience with using cp to copy 432 million files (39 TB)

#238

Post by dredmorbius »

pflanze wrote: post #232
dredmorbius wrote: post #231

The email states that file-based copy operations were used in favor of dd due to suspected block errors. Two questions come to mind: 1. I've not used dd on failing media, so I'm not sure of the behavior. Will it plow through a file with block-read failures or halt? 2. There's the ddrescue utility, which is specifically intended for reading from nonreliable storage. Seems that this could have offered another means for…

OP said "I went for a file-level copy because then I'd know which files contained the bad blocks". When you copy the block device with ddrescue (dd doesn't have logic to work around the bad sectors and the only sensible action for it is thus to stop, but don't take my word for it), the result will just have zeroes in the places where bad blocks were, and, assuming the filesystem structure is good enough (you should r…

I've played with ddrescue very lightly. From the GNU webpage linked above, it appears it creates logfiles which can be examined:

Ddrescuelog is a tool that manipulates ddrescue logfiles, shows logfile contents, converts logfiles to/from other formats, compares logfiles, tests rescue status, and can delete a logfile if the rescue is done. Ddrescuelog operations can be restricted to one or several parts of the logfile if the domain setting options are used.

That might allow for identification of files with bad sectors.

Re: My experience with using cp to copy 432 million files (39 TB)

#239

Post by fsniper »

rbh42 wrote: post #212
fsniper wrote: post #149

Earlier quoted context omitted.

Also there is no mention of unrecoverable file analysis like error handling of cp operations in the article. And with this many files it would not be feasible without using an error log file. So going with a simple block copy should suffice IMHO.

I'm the OP, so I can shred a bit of light on that: Dell's support suggested a file-level copy when I asked them what they recommended (but I'm not entirely sure they understood the implications). Also, time was not a big issue. I did keep a log file with the output from cp, and it clearly identified the filenames for the inodes with bad blocks. Actually, I'm not sure how dd would handle bad blocks.

Thank you for clarification.

I was about to bet on "read fail repeat skip" cycle for dd's behaviour but, looking into coreutil's source code at https://github.com/goj/coreutils/blob/master/src/dd.c , if I'm not mistaken , dd does not try to be intelligent and just uses a zeroed out buffer so It would return 0's for unreadable blocks.

Re: My experience with using cp to copy 432 million files (39 TB)

#240

Post by pflanze »

grondilu wrote: post #223
brazzy wrote: post #204

Earlier quoted context omitted.

The pipe is not the issue; you actually are storing the full paths of all files in the archive somewhere (namely in RAM) so that you're able to decide whether a given file you're looking at is an alternative hardlink for a file you have already sent. Yes, using tar for copying is not new and has its use cases, but this is not one of them.

Certainly not for all files. The filesystem keeps a track of the number of nodes links for each file. So tar does not have to memorize the inode for each file, but only for those who have duplicates. So unless a large amount of files have more than one hard link, this is probably not a problem. We will never know though, considering the author did not try it. Neither did he explain why he did not, despite the fact th…

From the original post: "We use rsnapshot (..) so most of the files have a high link count."

> it is the recommended Unix procedure

I don't think it is "recommended procedure", not today. At least on GNU/Linux cp is just fine. It may have been different two decades ago for some reason: I remember a graphical file manager (called "dfm") that used piped tar back in the late 90ies [1]. I know that installing open source utilities was pretty common back then to get better programs on closed source Unixes, and tar was probably high on the priorities to install whereas cp less so, and hence tar might have in fact been better than the preinstalled cp. I haven't seen anyone in the public technical (hence primarily open source) community advocating that even for portability reasons, so I can only think of it as "perhaps historically relevant, if at all".

Your point is valid though that one could have tested it here. See rbh42's answer on this point. Although I'm not sure he's right about both tar instances requiring a hash table: if tar submits the hard linking as from a previously submitted path to the new one, then there's no need for the table. And my quick look at the transmitted data seems to contain the source path for the link. The question then becomes whether tar has a better optimized hash table (not very likely), or uses a better algorithm (see my post about "megacopy" for an attempt at the latter).

[1] It actually had a problem: it didn't put "--" between the options and the path, and promptly failed when I moved an item that had a file name starting with a minus: it was interpreted by tar as option, and in addition the file manager didn't properly check the exit codes and simply unlinked the source even though it wasn't copied. The tools before GNU may really have been wanting: his failure to use "--" may have been caused by some contemporary non-GNU tar not supporting it. Anyway, of course I moved on to another file manager. Those bits of mine were sacrificed to the god of superior recommended Unix procedures or something.

Post reply on HN