Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
This only holds true so long as all data is on tape and you don't need a replica before sending it off, in case someone hits your petamobile. That's because your bandwidth is limited by your tape library, the drives, the network to your tape library, the latency of retrieving/writing/copying to tape, and few other things.
Having transferred petabytes of data in tens of millions of files over the past months let me assure you there's only one tool that you really need: GNU parallel. Whether you copy the individual files with ftp, scp or rsync is largely irrelevant. The network is always your ultimate bottleneck. Using a slower copy-tool just means having to set a slightly higher concurrency in order to max it out.
For bulk transfer of many files, and especially transfer over a local/nearby network that may be hold true, but as a general practice, and especially for serial/one off transfer of very large files, GNU parallel won't help. However, all the tools mentioned multiplex connections. They are better for transferring individual files, and you can also use them in parallel. A combination of your network, a CPU threads, fram…
For bulk transfer, the absolute fastest I've seen is piping tar through netcat and doing the reverse on the receiving end - on a 10-gigabit lan that results in transfer at the hdd speed. That was between my personal machines with consumer-grade SATA hard disks. The situation probably changes once you add hops and have multiple disks to read from at once.
If you have to regularly transfer large amounts of data over a network, it might be worth looking into a wan optimization product like Riverbed's Steelhead, Silverpeak's VX/NX lines, or Bluecoat Mach 5, or one of the other vendors' solutions. Yeah, you could try and roll it yourself, since really it just comes down to compressing and deduplicating what you send over the wire, but doing that well and also making it si…
There's not many situations where these types of products help, in my experience. Especially for the type of data that's going to be transferred between UCI and the Broad. Enterprise compute data, cached webpages, etc., may have a good amount of deduplication capacity. But for actual "data," being measurements etc, these products will achieve nothing. The data itself almost never has any duplicated chunks, and if the…
We had to explain this repeatedly to several vendors the last time we were buying a small-ish (30 TB) file server. They seemed very skeptical of this concept that we were storing lots of data in compressed binary formats.
If you have to regularly transfer large amounts of data over a network, it might be worth looking into a wan optimization product like Riverbed's Steelhead, Silverpeak's VX/NX lines, or Bluecoat Mach 5, or one of the other vendors' solutions. Yeah, you could try and roll it yourself, since really it just comes down to compressing and deduplicating what you send over the wire, but doing that well and also making it si…
There's not many situations where these types of products help, in my experience. Especially for the type of data that's going to be transferred between UCI and the Broad. Enterprise compute data, cached webpages, etc., may have a good amount of deduplication capacity. But for actual "data," being measurements etc, these products will achieve nothing. The data itself almost never has any duplicated chunks, and if the…
Yeah, they definitely won't do well over non-compressible non-repetitive data, but they can help for situations where the data isn't compressed at rest or where data is repetitive. Like I said, for many technical people you can roll your own and get reasonably close, but not everyone or every application that deals with large amounts of data fits that mold.
Having transferred petabytes of data in tens of millions of files over the past months let me assure you there's only one tool that you really need: GNU parallel. Whether you copy the individual files with ftp, scp or rsync is largely irrelevant. The network is always your ultimate bottleneck. Using a slower copy-tool just means having to set a slightly higher concurrency in order to max it out.
For bulk transfer of many files, and especially transfer over a local/nearby network that may be hold true, but as a general practice, and especially for serial/one off transfer of very large files, GNU parallel won't help. However, all the tools mentioned multiplex connections. They are better for transferring individual files, and you can also use them in parallel. A combination of your network, a CPU threads, fram…
This only holds true so long as all data is on tape and you don't need a replica before sending it off, in case someone hits your petamobile. That's because your bandwidth is limited by your tape library, the drives, the network to your tape library, the latency of retrieving/writing/copying to tape, and few other things.
That's why the proverbial "tapes" should not be actual tapes but full Hadoop nodes. Just unrack them and go (or ship an entire rack). http://research.microsoft.com/apps/pubs/default.aspx?id=6457...
That was 13 years ago. Seems like the economics have changed in favor of the internet by now.