Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving all your data, 9TB edition

about.gitlab.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Moving all your data, 9TB edition

#21

Post by zdw »

bluedino wrote: post #20
mattzito wrote: post #3

It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way). First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from…

Would that many rsyncs on the same volume thrash the disk like crazy?

Yes, it likely would - most disks will top out on read i/o rate, especially on somewhat random access.

rsync in general is quite optimized and usually the limiting factor to a data transfer is the network or disk rather than CPU speed (unless crypto is involved, for example, over a ssh connection)

Re: Moving all your data, 9TB edition

#22

Post by mattzito »

hawski wrote: post #12
mattzito wrote: post #9

Earlier quoted context omitted.

Which is how I've done it in the past - I'm sure these days there's utilities that will do it for you, but I had a bunch of perl code that would fork off N threads out of a queue and as one exited successfully, kick off another worker. The issue with xargs back in the day was that you might need to run several hundred rsync processes, and suddenly launching 500+ processes in parallel made your server very very sad. S…

$ man xargs --max-args=max-args -n max-args Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the -x option is given, in which case xargs will exit. ... --max-procs=max-procs -P max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use…

This was also...14?-ish years ago. Sometimes on Solaris 2.6 or 8 boxes. I am pretty sure xargs didn't have that flag back then (which is why I said, "back in the day").

Re: Moving all your data, 9TB edition

#23

Post by sytse »

cplease wrote: post #18

Honestly, this seems to me like a case study in getting carried away by cloud. There's a very strong case for cloud hosting when you have a clear need for elasticity and an distributed application to leverage it. Here, you have your entire business in one non-distributed Amazon instance. Amazon does not provide excellent service, availability, flexibility, or value for this model. It is in every way inferior to what…

We're pretty aware of the advantages and drawback of the cloud. We moved from AWS to a private server and then back to AWS. You're right that cloud hosting makes more sense if you need scaling. The move described was to a single server. But now that we have the data on AWS we can expand. Probably the first step will be a separate fileserver and appserver. After that we can add multiple appservers and consider availability zone failover.

Re: Moving all your data, 9TB edition

#24

Post by sytse »

ajays wrote: post #16
VikingCoder wrote: post #13

Earlier quoted context omitted.

Seriously. Why is this not discussed more? Seems obvious to me.

Maybe because the younguns today don't know what a "station wagon" is and what "tapes" are... ;-) (I jest)

And because an Amazon Import http://aws.amazon.com/importexport/ can take 3 weeks. No need to speed down the highway in that case :)

Re: Moving all your data, 9TB edition

#25

Post by cmurf »

I wonder how GlusterFS would perform in this case. Replicated volumes with one or more local nodes for the most likely failover requirement, with async-georeplication in case the building burned down.

Re: Moving all your data, 9TB edition

#27

Post by Aissen »

mattzito wrote: post #3

It seems to me that, in fact, your original idea was, in fact, the correct one - rsync probably would have been the best way to do this (and separately, a truck full of disks probably would have been the other best way). First, rsync took too long probably because you used just one thread and didn't optimize your command-line options - most of the performance problems with rsync with large filesystem trees comes from…

> (and separately, a truck full of disks probably would have been the other best way).

A truck ? I think two 3.5" 5TB hard drives would be enough, no ?

Re: Moving all your data, 9TB edition

#28

Post by VikingCoder »

sytse wrote: post #24
ajays wrote: post #16

Earlier quoted context omitted.

Maybe because the younguns today don't know what a "station wagon" is and what "tapes" are... ;-) (I jest)

And because an Amazon Import http://aws.amazon.com/importexport/ can take 3 weeks. No need to speed down the highway in that case :)

Wow, I had no idea. 3 weeks for them to do the physical hard drive hook-up, or 3 weeks of data transfer once it is hooked up? (Or a combo?)

Re: Moving all your data, 9TB edition

#29

Post by sytse »

VikingCoder wrote: post #28
sytse wrote: post #24

Earlier quoted context omitted.

And because an Amazon Import http://aws.amazon.com/importexport/ can take 3 weeks. No need to speed down the highway in that case :)

Wow, I had no idea. 3 weeks for them to do the physical hard drive hook-up, or 3 weeks of data transfer once it is hooked up? (Or a combo?)

I think there was a queue to do the hook-up work, not completely sure. Please check with Amazon when you need this, it might be less now.
Post reply on HN