21–27 of 27 posts
Having transferred petabytes of data in tens of millions of files over the past months let me assure you there's only one tool that you really need: GNU parallel. Whether you copy the individual files with ftp, scp or rsync is largely irrelevant. The network is always your ultimate bottleneck. Using a slower copy-tool just means having to set a slightly higher concurrency in order to max it out.
Having transferred petabytes of data in tens of millions of files over the past months let me assure you there's only one tool that you really need: GNU parallel. Whether you copy the individual files with ftp, scp or rsync is largely irrelevant. The network is always your ultimate bottleneck. Using a slower copy-tool just means having to set a slightly higher concurrency in order to max it out.
Forgive me if I'm being ignorant, but what's stopping me from making a large torrent?
Earlier quoted context omitted.
Ever tried to make a torrent containing millions of files? ;-)
No, but I do have tar -czf whatever.tar.gz, right?
I'm also rather skeptical that the common BT clients are made to handle files in the multi-terabyte range very well.
And finally, BT only makes sense when you're transferring to multiple destinations. There are better options for 1-to-1 transfers.
Earlier quoted context omitted.
No, but I do have tar -czf whatever.tar.gz, right?
We usually don't have the patience nor spare disk space to spend days or weeks on creating a multi-terabyte tar-archive first. I'm also rather skeptical that the common BT clients are made to handle files in the multi-terabyte range very well. And finally, BT only makes sense when you're transferring to multiple destinations. There are better options for 1-to-1 transfers.
Earlier quoted context omitted.
No, but I do have tar -czf whatever.tar.gz, right?
We usually don't have the patience nor spare disk space to spend days or weeks on creating a multi-terabyte tar-archive first. I'm also rather skeptical that the common BT clients are made to handle files in the multi-terabyte range very well. And finally, BT only makes sense when you're transferring to multiple destinations. There are better options for 1-to-1 transfers.
Earlier quoted context omitted.
There's not many situations where these types of products help, in my experience. Especially for the type of data that's going to be transferred between UCI and the Broad. Enterprise compute data, cached webpages, etc., may have a good amount of deduplication capacity. But for actual "data," being measurements etc, these products will achieve nothing. The data itself almost never has any duplicated chunks, and if the…
We had to explain this repeatedly to several vendors the last time we were buying a small-ish (30 TB) file server. They seemed very skeptical of this concept that we were storing lots of data in compressed binary formats.
As data science progresses, the amount of enterprisey large data situations will also decrease, I think.