These are the types of stories I love. I just learned a boat load in 5 minutes.
Is there maybe an archive website dedicated to these kind of stories?
If not, there should be.
My experience with using cp to copy 432 million files (39 TB)
71–80 of 267 posts
Re: My experience with using cp to copy 432 million files (39 TB)
#72> Wanting the buffers to be flushed so that I had a complete logfile, I gave cp more than a day to finish disassembling its hash table, before giving up and killing the process....Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. Does anyone know what the 'tear down' part is about? If it's about erasing the hashtable from memory, what takes so lo…
Looking at the code, it looks like deallocating a hash table requires traversing the entire table, because there is malloc()'d memory associated with each hash entry, so each entry has to be visited and free()'d. From hash_free() in coreutils hash.c: for (bucket = table->bucket; bucket bucket_limit; bucket++) { for (cursor = bucket->next; cursor; cursor = next) { next = cursor->next; free (cursor); } } Whereas if you…
Isn't it a terribly inefficient design to malloc hash entries individually? It would make more sense to use a pool / arena for that.
Re: My experience with using cp to copy 432 million files (39 TB)
#73The difficulty is that you are using a filesystem hierarchy to 'copy files' when you actually want to do a volume dump (block copy). Use XFS and xfsdump, or ZFS and zfs send, to achieve this.
Copy with hard link preservation is essentially like running dedupe except that you know ahead of time how many dupes there are. Dedupe is often very memory intensive, and even well thought out implementations don't support keeping book keeping structures on disk.
Re: My experience with using cp to copy 432 million files (39 TB)
#74Earlier quoted context omitted.
Looking at the code, it looks like deallocating a hash table requires traversing the entire table, because there is malloc()'d memory associated with each hash entry, so each entry has to be visited and free()'d. From hash_free() in coreutils hash.c: for (bucket = table->bucket; bucket bucket_limit; bucket++) { for (cursor = bucket->next; cursor; cursor = next) { next = cursor->next; free (cursor); } } Whereas if you…
Why exactly is it necessary to to free each hash entry instead of exiting the process?
For one it makes automated leak detection tools like Valgrind useless.
Re: My experience with using cp to copy 432 million files (39 TB)
#75Earlier quoted context omitted.
Looking at the code, it looks like deallocating a hash table requires traversing the entire table, because there is malloc()'d memory associated with each hash entry, so each entry has to be visited and free()'d. From hash_free() in coreutils hash.c: for (bucket = table->bucket; bucket bucket_limit; bucket++) { for (cursor = bucket->next; cursor; cursor = next) { next = cursor->next; free (cursor); } } Whereas if you…
Why exactly is it necessary to to free each hash entry instead of exiting the process?
Others have already mentioned that it isn't really necessary, but for some historical context this wasn't always true. When I first started programming computers, apps (written in languages that aren't garbage collected) on your typical consumer level computer/OS had to be really careful with freeing up all allocated memory before exiting or that memory would be gone forever (as far as the OS was concerned when it came to reallocating it to other apps) until the user rebooted the computer.
Advances coming from the combination of more advanced operating systems and CPU features like MMUs have made this a non-issue in most cases (can still be an issue on embedded, etc).
Re: My experience with using cp to copy 432 million files (39 TB)
#76This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…
Wouldn't rsync of been a better and more reliable choice for this?
s/of/have
Re: My experience with using cp to copy 432 million files (39 TB)
#77So it was all the files in one go, presumably with `cp -r`? What about doing something with find/xargs/i-dunno to copy all the files, but break em into batches so you aren't asking cp to do it's bookkeeping for so many files in one process? Would that work better? Or worse in other ways?
This page may be useful:
http://unix.stackexchange.com/questions/44247/how-to-copy-di...
The main issue is that there's no api to get the list of files hard linked together: the only way is to check all the existing files and compare inodes. If you're doing a plain copy over 2 fs, you cannot choose which number the target inode will be, so you need to keep a map between inode numbers, or between inodes and file names ("cp" does the later).
Re: My experience with using cp to copy 432 million files (39 TB)
#78The difficulty is that you are using a filesystem hierarchy to 'copy files' when you actually want to do a volume dump (block copy). Use XFS and xfsdump, or ZFS and zfs send, to achieve this. Copy with hard link preservation is essentially like running dedupe except that you know ahead of time how many dupes there are. Dedupe is often very memory intensive, and even well thought out implementations don't support keep…
"Normally I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained the bad blocks."
Re: My experience with using cp to copy 432 million files (39 TB)
#79Re: My experience with using cp to copy 432 million files (39 TB)
#80Earlier quoted context omitted.
Why exactly is it necessary to to free each hash entry instead of exiting the process?
If it's the last thing you do before you exit the process, it isn't necessary, because the OS will reclaim your process's memory in one fell swoop. I believe that's what the linked post is advocating 'cp' should do. (At least on modern systems that's true; maybe there are some exotic old systems where not freeing your data structures before exit causes permanent memory leaks?) It's seen as good C programming practice…
I tend to do this in my C programs because in development usually have malloc() wrapped so that if any block hasn't been free()'ed it's reported at exit() time. This kind of check for lost pointers is usually so cheap that you use it even if you never expect to run on a system without decent memory management.
As an aside, GNU libc keeps ( or at least used to keep, I haven't checked in years ) the pointers used by malloc()/free() next to the blocks themselves, which gives really bad behavior when freeing a large number of blocks that have been pushed out to swap--you wind up bringing in pages in order to free them because the memory manager's working set is the size of all allocated memory. Years ago I wrote a replacement that avoided this just to speed up Netscape's horrible performance when it re-sized the bdb1.85 databases it used to track browser history. The browser would just "go away" thrashing the disk for hours and killing it just returned you to a state where it would decide to resize again an hour or so after a restart. Using LD_PRELOAD to use a malloc that kept it's bookkeeping away from the allocated blocks changed hours to seconds.