Moving all your data, 9TB edition
about.gitlab.com
Moving all your data, 9TB edition
1–10 of 32 posts
Re: Moving all your data, 9TB edition
#2The two lessons to take away from this:
1. Ask someone else who has already done what you're thinking of doing. They have already made all the mistakes you might make and have figured out a way that works.
2. Assume that whatever you think will work will fail in an unexpected way with probably-catastrophic results. Test everything before you try something new.
Re: Moving all your data, 9TB edition
#3It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way).
First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from using one command to run everything, something like:
rsync -av /source/giant/tree /dest/giant/tree
And the process of crawling, checksumming, storing is not only generally slow, but incredibly inefficient on today's modern multicore processors.
Much better to break it up into many threads, something like:
rsync -av /source/giant/tree/subdir1 /dest/giant/tree/subdir1
rsync -av /source/giant/tree/subdir2 /dest/giant/tree/subdir2
rsync -av /source/giant/tree/subdir3 /dest/giant/tree/subdir3
That alone probably would have dramatically sped things up, BUT you do still have your speed of light issues.
This is where Amazon import/export comes in - do a one-time tar/rsync of your data to an external 9TB array, ship it to Amazon, have them import it to S3, load it onto your local Amazon machines.
You now have two copies of your data - one on s3, and one on your amazon machine.
Then you use your optimized rsync to run and bring it up to a relatively consistent state - i.e. it runs for 8 hours to sync up, now you're 8 hours behind.
Then you take a brief downtime and run the optimized rsync one more time, and now you have two fully consistent filesystems.
No need for drbd and all the rest of this - just rsync and an external array.
I've used this method to duplicate terabytes and terabytes of data around, and 10s of millions of small files. It works, and is a lot fewer moving parts than drbd
Re: Moving all your data, 9TB edition
#4It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way). First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from…
Whenever someone gripes that rsync/scp is slow, it's usually because they didn't bother to look into proper solutions for their problem. rsync will barely fill the pipe in many/most cases. Using GridFTP or bbcp is usually preferred.
Re: Moving all your data, 9TB edition
#5It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way). First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from…
Thanks for the suggestions. Amazon import is great but we were informed they needed 3 weeks to perform the import, we didn't have time for that.
Re: Moving all your data, 9TB edition
#6Stories like these remind me of Andy Tannenbaum's statement: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
Re: Moving all your data, 9TB edition
#7It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way). First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from…
Whenever someone gripes that rsync/scp is slow, it's usually because they didn't bother to look into proper solutions for their problem. rsync will barely fill the pipe in many/most cases. Using GridFTP or bbcp is usually preferred.
You can also do well relying on the OS scheduler and networking stack by forking off many rsync processes using GNU parallel or xargs.
Re: Moving all your data, 9TB edition
#8It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way). First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from…
Thanks for the suggestions. Amazon import is great but we were informed they needed 3 weeks to perform the import, we didn't have time for that.
> we were able to move a 9TB filesystem to a different data center and hosting provider in three weeks
But it took you 3 weeks anyway?
Re: Moving all your data, 9TB edition
#9Earlier quoted context omitted.
Whenever someone gripes that rsync/scp is slow, it's usually because they didn't bother to look into proper solutions for their problem. rsync will barely fill the pipe in many/most cases. Using GridFTP or bbcp is usually preferred.
You can also do well relying on the OS scheduler and networking stack by forking off many rsync processes using GNU parallel or xargs.
Which is how I've done it in the past - I'm sure these days there's utilities that will do it for you, but I had a bunch of perl code that would fork off N threads out of a queue and as one exited successfully, kick off another worker.
The issue with xargs back in the day was that you might need to run several hundred rsync processes, and suddenly launching 500+ processes in parallel made your server very very sad. So you needed some basic job queueing system.
Re: Moving all your data, 9TB edition
#10I love these kinds of write-ups, because I always learn new techniques and tools.
I've done this sort of thing with rsync, and with ZFS send/receive.
And of course, mailing hard disks.