11–20 of 267 posts
I would probably have used tar|tar for this, or rsync.
I would probably have used tar|tar for this, or rsync.
Why?
I feel the pain. I went thru the same hell a few months ago.
> I browsed the net for other peoples' experience with copying many files and quickly decided that cp would do the job nicely.
After 20 years you no longer google how to copy files.
Edit: Reading on he talks about strace and even reading cp's source code which makes it even weirder that he had to google how to do this...
Edit2: Comments! Took only ten downvotes before someone bothered to explain what I was doing wrong, but now there are three almost simultaneously. I guess those make a few good points. I'd still think cp ought to handle just about anything especially given its ubiquitousness and age, but I see the point.
And to clarify: I'm not saying the author is stupid or anything. It's just weird to me that someone with that much experience would google something which on the surface sounds so trivial, even at 40TB.
I would probably have used tar|tar for this, or rsync.
Rsync has similar issues when you are copying hard links. It has to keep a table of all the inodes/filename pairs it sees so it can detect hard links and relink them on the destination. This can be big if you have a ton of hard links (say in a backup situation). Rsync at least acts reasonably idempotent, so you can just run it again if it gets interrupted, which is usually why I use it for large copies. I don't remem…
--partial keep partially transferred files
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
I would probably have used tar|tar for this, or rsync.
For future reference for other folks, the command would look something like this:
cd /old-directory && tar czvflpS - . | tar -C /new-directory -xzvf -
cd /old-directory && tar czvflpS - . | ssh -i /path/to/private-key user@host "tar -C /new-directory -xzvf -"
Disk-to-disk SATA or SAS without ssh in between would be significantly faster.