Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

181–190 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#181

Post by andreasvc »

steveh73 wrote: post #78
angry_octet wrote: post #73

The difficulty is that you are using a filesystem hierarchy to 'copy files' when you actually want to do a volume dump (block copy). Use XFS and xfsdump, or ZFS and zfs send, to achieve this. Copy with hard link preservation is essentially like running dedupe except that you know ahead of time how many dupes there are. Dedupe is often very memory intensive, and even well thought out implementations don't support keep…

"Normally I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained the bad blocks."

Given the issues encountered it would make sense to split this into two passes: 1. identify files with bad blocks and log; 2. do the block-level copy. The disadvantages of file-level copy weigh heavier than the doing this in a single step.

Re: My experience with using cp to copy 432 million files (39 TB)

#182

Post by laichzeit0 »

ritchiea wrote: post #43
mjn wrote: post #34

Earlier quoted context omitted.

Looking at the code, it looks like deallocating a hash table requires traversing the entire table, because there is malloc()'d memory associated with each hash entry, so each entry has to be visited and free()'d. From hash_free() in coreutils hash.c: for (bucket = table->bucket; bucket bucket_limit; bucket++) { for (cursor = bucket->next; cursor; cursor = next) { next = cursor->next; free (cursor); } } Whereas if you…

Why exactly is it necessary to to free each hash entry instead of exiting the process?

As others have commented, it's not necessary.

If you wrote cp in such a way that it's a reusable C library that other programs might use, then I can understand why they do this. It still doesn't make much sense for the command-line version to call this. Of course, it might run on some weird platform where the OS doesn't reclaim the memory of a process on exit(?).

Re: My experience with using cp to copy 432 million files (39 TB)

#183

Post by ygra »

ccozan wrote: post #175
brazzy wrote: post #168

Earlier quoted context omitted.

Would have been even worse in this case due to the requirement to preserve hard links.

tar can preserve hardlinks. In fact, tar is really the best solution in this case, using the streaming between two tar commands.

Wouldn't it somehow have to keep about the same data as cp for preserving hard links?

Re: My experience with using cp to copy 432 million files (39 TB)

#184

Post by noselasd »

gpvos wrote: post #159
derefr wrote: post #92

Earlier quoted context omitted.

If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined. Plan9's default filesystem is made up of two parts: Fossil, and Venti. - Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values. - Venti is a persistent graph database that holds what would today be called "inode met…

For my understanding: what happens if you open a file, change one byte and close it again? Since the SHAsum of the contents has changed, is the entire file now copied?

Only the block where the byte resides. A block is typically 512b to 4096b. (So it's not that unlike a normal drive, where you also have to rewrite an entire sector even if just a byte changed)

Venti doesn't know about files, it only knows about blocks of data. It's a key/value store where the key is sha-1 and the value is a block(blob) of data.

The filesystem running on top of Venti will ask Venti to store the new block where you changed the byte and the filesystem will update the metadata that assembles all blocks to a file.

Re: My experience with using cp to copy 432 million files (39 TB)

#185

Post by asveikau »

pedrocr wrote: post #27

Unix could really use a way to get all the paths that point to a given inode. These days that shouldn't really cost all that much and this issue comes up a lot in copying/sync situations. Here's the git-annex bug report about this: https://git-annex.branchable.com/bugs/Hard_links_not_synced_...

Wow, it's not every day I hear about a filesystem feature that Windows has and Linux doesn't. (On a recent windows system: fsutil hardlink list -- you can try any random exe or DLL in system32 for an example of a hard link.)

I forget what the api for that looks like if I ever knew. Might be private.

I am surprised, usually Linux is way ahead of Windows on shiny filesystem stuff.

Re: My experience with using cp to copy 432 million files (39 TB)

#186

Post by noselasd »

pedrocr wrote: post #98
derefr wrote: post #92

Earlier quoted context omitted.

If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined. Plan9's default filesystem is made up of two parts: Fossil, and Venti. - Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values. - Venti is a persistent graph database that holds what would today be called "inode met…

> Honestly, I'm terribly confused why all filesystems haven't been broken into these two easily-separable layers. Is it just inertia? The penalty for doing content addressed filesystems is of course the CPU usage. btrfs probably has most of the benefits without the CPU cost with its copy-on-write semantics. Note that what you describe (and my initial process) is a different semantic than hard-links. What you get is s…

> Honestly, I'm terribly confused why all filesystems haven't been broken into these two easily-separable layers. Is it just inertia

They are. Pretty much all operating systems provides a block layer, on top of which filesystems are normally layered.

Though the block layer isn't content addressed, it's just indexed (store block number 55, read block number 9. etc.)

Re: My experience with using cp to copy 432 million files (39 TB)

#187

Post by cnvogel »

dbbolton wrote: post #13

>We use XFS Why?

I personally still consider XFS a very mature and reliable filesystem. Both in terms of utility programs and kernel implementation. If I remember correctly, it was ported to linux from SGI/Irix where it was used for decades. It also was the default fs for RedHat/centos for a long time, so it might still have stuck at many shops.

Heres my anecdotal datapoint on which I base my personal believe:

From about 10-6 years ago, when I was doing sysadmin-work at university building storage-systems from commodity parts for experimental bulk data, we first had a load of not-reliably working early raid/sata(?) adapters, and those made ext3 and reiserfs (I think...) oops the kernel when the on-disk structure went bad. Whereas XFS just put a "XFS: remounted FS readonly due to errors" in the kernel logfile. That experience made XFS my default filesystem up to recently when I started to switch to btrfs. (of course, we fixed the hardware-errors, too... :-) )

Also, from that time, I got to use xfsdump/xfsrestore for backups and storage of fs-images which not even once failed on me.

Re: My experience with using cp to copy 432 million files (39 TB)

#188

Post by nailer »

sitkack wrote: post #21
0x0 wrote: post #5

I wonder how well rsync would have fared here.

Rsync can die just from scanning the whole directory tree of files first.

If it's changing? You'd snapshot (assuming either LVM or BtrFS) then rsync of the snapshot.

Re: My experience with using cp to copy 432 million files (39 TB)

#189

Post by brazzy »

ccozan wrote: post #175
brazzy wrote: post #168

Earlier quoted context omitted.

Would have been even worse in this case due to the requirement to preserve hard links.

tar can preserve hardlinks. In fact, tar is really the best solution in this case, using the streaming between two tar commands.

As discussed in detail elswhere abive: the problem is not with the ability to preserve hardlinks, but the resources, namely RAM, required to do so.

Two tar command would in fact be much worse, because they'd require twice as many resources.

Re: My experience with using cp to copy 432 million files (39 TB)

#190

Post by grondilu »

brazzy wrote: post #189
ccozan wrote: post #175

Earlier quoted context omitted.

tar can preserve hardlinks. In fact, tar is really the best solution in this case, using the streaming between two tar commands.

As discussed in detail elswhere abive: the problem is not with the ability to preserve hardlinks, but the resources , namely RAM, required to do so. Two tar command would in fact be much worse, because they'd require twice as many resources.

I doubt it, since you're using a pipe. It's not like you're actually storing the whole archive anywhere.

Also, and just to be clear, I did not invent this idea of using tar for this. I remember having read it in a Unix manual when I was learning about this OS.

And even if using tar is not a good idea, it should certainly be considered and as such I don't understand why OP doesn't even mention it in his post.

Post reply on HN