51–60 of 267 posts
This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild…
How does the rebuild time of ZFS's raidz compare to RAID5/6?
Unix could really use a way to get all the paths that point to a given inode. These days that shouldn't really cost all that much and this issue comes up a lot in copying/sync situations. Here's the git-annex bug report about this: https://git-annex.branchable.com/bugs/Hard_links_not_synced_...
https://sort.symantec.com/public/documents/sfha/6.0/linux/pr... http://sfdoccentral.symantec.com/sf/5.0/aix/manpages/vxfs/vx...
I wonder if any modern filesystems (xfs/ext4/btrfs/zfs) can actually enable tracking this.
Earlier quoted context omitted.
Looking at the code, it looks like deallocating a hash table requires traversing the entire table, because there is malloc()'d memory associated with each hash entry, so each entry has to be visited and free()'d. From hash_free() in coreutils hash.c: for (bucket = table->bucket; bucket bucket_limit; bucket++) { for (cursor = bucket->next; cursor; cursor = next) { next = cursor->next; free (cursor); } } Whereas if you…
Why exactly is it necessary to to free each hash entry instead of exiting the process?
Obviously many programmers do not think to (or cannot easily) test their code with 40 TBs of data.
Interesting. In Windows-land, the default copy is pretty anemic, so probably most people avoid it for serious work. I'd probably use robocopy from the command line. And if I was being lazy, I'd use the Teracopy GUI. I think my limit for a single copy command has been around 4TB with robocopy--and that was a bunch of large media files, instead of smaller more numerous files. Maybe there's a limit I haven't hit.
So it was all the files in one go, presumably with `cp -r`? What about doing something with find/xargs/i-dunno to copy all the files, but break em into batches so you aren't asking cp to do it's bookkeeping for so many files in one process? Would that work better? Or worse in other ways?
Earlier quoted context omitted.
You're right to recommend a tarpipe. I've had to copy several very large BackupPC storage pools in the past, and a tarpipe is the most reliable way to do it. (The only downside to BackupPC IMO...) For future reference for other folks, the command would look something like this: cd /old-directory && tar czvflpS - . | tar -C /new-directory -xzvf - Tarpipes are especially neat because they can work well over ssh (make s…
What's the benefit of using a tarpipe locally?
So it was all the files in one go, presumably with `cp -r`? What about doing something with find/xargs/i-dunno to copy all the files, but break em into batches so you aren't asking cp to do it's bookkeeping for so many files in one process? Would that work better? Or worse in other ways?
I would probably have used tar|tar for this, or rsync.
I'm unfamiliar with tar|tar, can you provide some incite into this trick?
Basically the same as tar's standard usage, except you don't store that stream as a tar file/image anywhere.