11–20 of 32 posts
Earlier quoted context omitted.
Thanks for the suggestions. Amazon import is great but we were informed they needed 3 weeks to perform the import, we didn't have time for that.
> we were able to move a 9TB filesystem to a different data center and hosting provider in three weeks But it took you 3 weeks anyway?
Earlier quoted context omitted.
You can also do well relying on the OS scheduler and networking stack by forking off many rsync processes using GNU parallel or xargs.
Which is how I've done it in the past - I'm sure these days there's utilities that will do it for you, but I had a bunch of perl code that would fork off N threads out of a queue and as one exited successfully, kick off another worker. The issue with xargs back in the day was that you might need to run several hundred rsync processes, and suddenly launching 500+ processes in parallel made your server very very sad. S…
--max-args=max-args
-n max-args
Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the -x option is given, in which case xargs will exit.
...
--max-procs=max-procs
-P max-procs
Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option with -P; otherwise chances are that only one exec will be done.
Stories like these remind me of Andy Tannenbaum's statement: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way). First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from…
That way instead of scanning the whole file system you just rsync the files that have changed.
[1] Assuming you can't hook into the app(s) making the changed directly. You can even just look for new/changed files if deletions are not a priority.
Stories like these remind me of Andy Tannenbaum's statement: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
Seriously. Why is this not discussed more? Seems obvious to me.
(I jest)
Here, you have your entire business in one non-distributed Amazon instance. Amazon does not provide excellent service, availability, flexibility, or value for this model. It is in every way inferior to what you would get from colo or managed dedicated server hosting. Hosting your whole business on a single anonymous Amazon cloud instance that you can't walk up to and touch is engineering malpractice.
It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way). First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from…