Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

191–200 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#191

Post by ygra »

asveikau wrote: post #185
pedrocr wrote: post #27

Unix could really use a way to get all the paths that point to a given inode. These days that shouldn't really cost all that much and this issue comes up a lot in copying/sync situations. Here's the git-annex bug report about this: https://git-annex.branchable.com/bugs/Hard_links_not_synced_...

Wow, it's not every day I hear about a filesystem feature that Windows has and Linux doesn't. (On a recent windows system: fsutil hardlink list -- you can try any random exe or DLL in system32 for an example of a hard link.) I forget what the api for that looks like if I ever knew. Might be private. I am surprised, usually Linux is way ahead of Windows on shiny filesystem stuff.

Is that really a feature the file system provides? Or does that command simply walk the MFT and finds all hardlinks to the same file record?

As far as I know NTFS does not store what hard links point to a file.

Re: My experience with using cp to copy 432 million files (39 TB)

#192

Post by mnw21cam »

georgemcbay wrote: post #75
ritchiea wrote: post #43

Earlier quoted context omitted.

Why exactly is it necessary to to free each hash entry instead of exiting the process?

Others have already mentioned that it isn't really necessary, but for some historical context this wasn't always true. When I first started programming computers, apps (written in languages that aren't garbage collected) on your typical consumer level computer/OS had to be really careful with freeing up all allocated memory before exiting or that memory would be gone forever (as far as the OS was concerned when it ca…

This was a particular problem on the Amiga system, among others. There were attempts to retro-fit resource tracking to the OS, but this was made all-but-impossible by the fact that a process can malloc() a chunk of memory, write a message in it, and then hand a pointer to that message to another process, expecting the other process to dealloc() it. This made for wonderfully cheap inter-process communication (for instance, the filesystem would load blocks from disc and pass them as messages to the process wanting them).

Re: My experience with using cp to copy 432 million files (39 TB)

#193

Post by saalweachter »

derefr wrote: post #92
pedrocr wrote: post #61

How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all…

If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined. Plan9's default filesystem is made up of two parts: Fossil, and Venti. - Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values. - Venti is a persistent graph database that holds what would today be called "inode met…

ZFS sits on top of zpools.

Re: My experience with using cp to copy 432 million files (39 TB)

#194

Post by pepve »

LeoPanthera wrote: post #47
thaumaturgy wrote: post #19

Earlier quoted context omitted.

You're right to recommend a tarpipe. I've had to copy several very large BackupPC storage pools in the past, and a tarpipe is the most reliable way to do it. (The only downside to BackupPC IMO...) For future reference for other folks, the command would look something like this: cd /old-directory && tar czvflpS - . | tar -C /new-directory -xzvf - Tarpipes are especially neat because they can work well over ssh (make s…

The prompt goes to stderr, the pipe only pipes stdout, so a prompt should not cause excessive bonage, as long as you're there to respond to it. Also, don't use -z locally, or even over a moderately fast network. The compression is not that fast and almost always makes things slower.

On modereately fast networks, instead of -z which uses a single threaded gzip implementation, parallel gzip may still give big improvements: http://zlib.net/pigz/

Re: My experience with using cp to copy 432 million files (39 TB)

#195

Post by yungchin »

theworst wrote: post #154
ori_b wrote: post #136

Earlier quoted context omitted.

You seem to be confusing venti and fossil.

Can you explain further? I am not a plan9 expert, by any means, but I'm stuck at where GP made the confusion. Thanks!

He just swapped the names I think - Venti is the block store, Fossil is the file system layer.

Re: My experience with using cp to copy 432 million files (39 TB)

#196

Post by spydum »

KaiserPro wrote: post #179
golan wrote: post #89

Earlier quoted context omitted.

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

WE use 10 rsyncs in parallel to copy .5pb in less than 3 days. CPU is cheap.

Have done the same, found that rsync seems to not natively parallelize itself, so spread across 20 cores, it really screamed.

Re: My experience with using cp to copy 432 million files (39 TB)

#197

Post by lambda »

lamontcg wrote: post #104
golan wrote: post #89

Earlier quoted context omitted.

The problem with rsync is that it would have to traverse all the FS tree and check every single file on both sides for the timestamp and the file. In a relatively small FS tree is just fine, but when you start having GBs and GBs and tens of thousands of files, it becomes somewhat impractical. Then, you need to come up with other creative solutions, like deep syncing inside the FS tree, etc. Fun times. Add checksummin…

if you want to copy everything and there's nothing at the target: rsync --whole-file --ignore-times that should turn off the metadata checks and the rsync block checksum algorithm entirely and transfer all of the bits at the source to the dest without any rsync CPU penalty. also for this purpose it looks like -H is also required to preserve hard links which the man page notes: "Note that -a does not preserve hardlink…

Yep, my answer when cp or scp is not working well is always to break out rsync, even if I'm just copying files over to an empty location.

I've had good luck with the -H option, though it is slower than without the option. I have never copied a filesystem with nearly as many files as the OP with -H; the most I've done is probably a couple million, consisting of 20 or so hardlinked snapshots with probably 100,000 or 200,000 files each. rsync -H works fine for that kind of workload.

Re: My experience with using cp to copy 432 million files (39 TB)

#198

Post by sophacles »

icedchai wrote: post #108
ajross wrote: post #86

Earlier quoted context omitted.

And hopefully you would have written up a similar essay on the oddball experiences you had with rsync, which is even more stateful than cp and even more likely to have odd interactions when used outside its comfort zone. Ditto for tricks like:(cd $src; tar cf - .) | (cd $dst; tar xf -). Pretty much nothing is going to work in an obvious way in a regime like this. That's sort of the point of the article.

Or maybe not. He mentions rsnapshot in the article, which uses rsync under the hood. This implies rsync would have a very good chance of handling a large number of hardlinks... since it created them in the first place.

That doesn't follow. If backups are for multiple machines to a big file server, the backup machine will have a much larger set of files than those that come from an individual machine. Further, each backup "image" compares the directory for the previous backup to the current live system. Generally it looks something like this:

1. Initial backup or "full backup" - copy the full targeted filesystem to the time indexed directory of the backup machine.

2. Sequential backups:

a. on the backup machine, create a directory for the new time, create a mirror directory structure of the previous time.

b. hard link the files in the new structure to those in the previous backup (which may be links themselves, back to the last full backup.

c. rsync the files to the new backup directory. Anything that needs to be transfered results in rsync transfering the file to a new directory, the moving it into the proper place. This unlinks the filename from the previous version and replaces it with the full version.

So yeah, the result of this system over a few machines and a long-timeframe backup system is way more links on the backup machine than any iteration of the backup will ever actually use.

Re: My experience with using cp to copy 432 million files (39 TB)

#199

Post by novaleaf »

fintler wrote: post #93

I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/h…

man! and here I am feeling like a champ conquering NTFS's long file name limitations :/

Re: My experience with using cp to copy 432 million files (39 TB)

#200

Post by peterwwillis »

72deluxe wrote: post #165
squiguy7 wrote: post #140

Earlier quoted context omitted.

"due to being jumbled with javascript..." - This made my day.

It's also an astute observation that is incredibly true! Most sites now incorporate JavaScript. I remember when it was "new-fangled" (and DHTML anyone?) and everyone put those annoying cursor trackers that trailed blobs from where your cursor was.

Uh oh, is it nostalgia time already?

What I mainly remember is two things: suddenly everyone had to do client-side form checking to prevent people from typing dashes in a phone number field, and everyone tried to imitate Java without actually using Java. It seems like any way you could enhance a page by combining mouseover events, gifs and loading content dynamically you would throw into a page, even if it made the user experience a total mess. Which of course never happens today.

My favorite memory is helping to write some of the first "griefers" using JS. If you were unlucky enough to stumble onto our page you had a window literally jumping around your screen with pop-ups til the end of time. There was no way to exit the browser until you'd clicked it all and it'd never end. Just getting your mouse near the 'X' would make the window jump around again. Of course, that was a mild annoyance compared to loading fonts and images so big they'd consume all the available RAM and make the machine fall over from swapping in about 15 seconds.

Post reply on HN