Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

61–70 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#61

Post by pedrocr »

How about this for a better cp strategy to deal with hardlinks:

1. Calculate the hash of /sourcedir/some/path/to/file

2. Copy the file to /tempdir/$hash if it doesn't exist yet

3. Hard-link /destdir/some/path/to/file to /tempdir/$hash

4. Repeat until you run out of source files

5. Recursively delete /tempdir/

This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all the hashing. If you're smart about doing steps 1 and 2 together it shouldn't require any additional I/O (ignoring the extra file metadata).

Edit: actually this won't recreate the same hardlink structure, it will deduplicate any identical files, which may not be what you want. Replacing the hashing with looking up the inode with stat() would actually do the right thing. And that would basically be an on-disk implementation of the hash table cp is setting up in memory.

Re: My experience with using cp to copy 432 million files (39 TB)

#62

Post by Nelson69 »

gwern wrote: post #26

> Wanting the buffers to be flushed so that I had a complete logfile, I gave cp more than a day to finish disassembling its hash table, before giving up and killing the process....Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. Does anyone know what the 'tear down' part is about? If it's about erasing the hashtable from memory, what takes so lo…

So it's a hash table that's about twice the size of physical memory. The entries in the hashtable are usually pointers to memory that you need to deallocate (char* filenames in this case, maybe some other stuff too) so if you're a good citizen you iterate through them all and deallocate (mark is as free) them as you go. It would seem there were some pathological swap issues doing that. It would be interesting to see if a tree behaved better in this type of situation. The allocator has tables of its own in memory too, if you have to swap those out to find the memory that your then going to deallocate then swap out all the user data structures to swap in the allocator's structures to do the deallocation or something whacky like that, it kind of makes sense.

His patch just skips that step and ends the program.

Re: My experience with using cp to copy 432 million files (39 TB)

#63

Post by hyperpape »

lucb1e wrote: post #16

> 20 years experience with various Unix variants > I browsed the net for other peoples' experience with copying many files and quickly decided that cp would do the job nicely. After 20 years you no longer google how to copy files. Edit: Reading on he talks about strace and even reading cp's source code which makes it even weirder that he had to google how to do this... Edit2: Comments! Took only ten downvotes before…

Whenever you get to extreme cases, there's always going to be something subtle. I'm sure he didn't google " how to copy files", but looked for people who'd worked with many terabytes of data. Reminds me of a nice story by Reginakd Braithwaite: http://raganwald.com/2013/03/26/the-interview.html

Re: My experience with using cp to copy 432 million files (39 TB)

#64

Post by fiatmoney »

LeoPanthera wrote: post #40
mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

How does the rebuild time of ZFS's raidz compare to RAID5/6?

It is generally superior because it can skip blank blocks. RAID is byte-by-byte.

Re: My experience with using cp to copy 432 million files (39 TB)

#65

Post by gizmo686 »

MichaelGG wrote: post #45
jrochkind1 wrote: post #28

Earlier quoted context omitted.

I don't understand it, but the OP says this, implying the author agrees with you for any modern system, but not on 'old systems without working memory management' > And unless old systems without working memory management must be supported, I don't see any harm in simply removing the call to the forget_all function towards the end of cp.c.

How would such a system even work? Any abnormal process termination or bugs would mean eventually the system becomes unusable.

The system would slowly leak memory until you reboot.

Re: My experience with using cp to copy 432 million files (39 TB)

#66

Post by Sanddancer »

dbbolton wrote: post #13

>We use XFS Why?

XFS really shines when you're dealing with large volumes, especially with RAID. There are certain features, like striped allocation, that mean you end up getting faster performance, because the filesystem has a better idea of how blocks are laid out on the disks.

Re: My experience with using cp to copy 432 million files (39 TB)

#67

Post by ntsb »

mililani wrote: post #25

This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…

Wouldn't rsync of been a better and more reliable choice for this?

Re: My experience with using cp to copy 432 million files (39 TB)

#68

Post by gizmo686 »

noinsight wrote: post #42
IvyMike wrote: post #10

Interesting. In Windows-land, the default copy is pretty anemic, so probably most people avoid it for serious work. I'd probably use robocopy from the command line. And if I was being lazy, I'd use the Teracopy GUI. I think my limit for a single copy command has been around 4TB with robocopy--and that was a bunch of large media files, instead of smaller more numerous files. Maybe there's a limit I haven't hit.

> Teracopy I've used FastCopy for GUI based larger transfers, it's open source and can handle larger datasets well in my experience. It also doesn't choke on >MAX_PATH paths. Haven't had problems with it. Supposedly it's the fastest tool around... The only slight issue is that the author is Japanese so the English translations aren't perfect plus the comments in the source are in Japanese.

">MAX_PATH paths"

How does this happen?

Re: My experience with using cp to copy 432 million files (39 TB)

#69

Post by ars »

kazinator wrote: post #8
mturmon wrote: post #6

Earlier quoted context omitted.

About rsync: if you just use -a, it does not copy hard links correctly: -a, --archive [...] Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H. If you do specify -H, rsync does keep track of hard links, but presumably at the cost of keeping a data structure similar to cp: -H, --hard-links This tells rsync to look for hard-linked files in the so…

Darn, thanks! I've been using rsync for incremental backups (themselves hard-link-based) for years, but neglecting to do this. That is to say, snapshot N+1 shares structure with backup N to save space (via hard links) via --link-dest. But files may be replicated because -H (missing) is orthogonal of --link-dest.

You can use fdupes (or various others similar to it) to find and automatically hardlink the duplicates.
Post reply on HN