21–30 of 35 posts
btw not relevant in your case, but a friend working for a large video company regularly drives a truckload of 10 TB tapes around the continent - bandwidth is still an issue at that scale.
btw not relevant in your case, but a friend working for a large video company regularly drives a truckload of 10 TB tapes around the continent - bandwidth is still an issue at that scale.
Amazing. If only there was away to get around the limitations at that level.
* Are we talking a few giant files or thousand of small-ish files?
* Who is on the receiving end? Technical people from who we can expect that they are able to run some command-line stuff or your average joe?
* What type of scenarios must the solution work? What OSes? Is it acceptable to install extra software or must it work out of the box?
etc. etc. etc.
If, for some reason, you are averse to rsync (there are plenty of Windows clients), then bittorrent is second-best: it works just as well for transferring private data, just turn off all the metadata broadcast options and make sure that encryption is required. (Not so easy for incremental transfers, though)
We considered rsync but were wondering if there were more specialized tools available. We figured that those who work in the scientific community would have a way to transfer their large data sets between institutions.
Scientific institutions that need to transfer large data sets have fast connections. :) How does 340 Gbps sound? Check out ESnet. http://newscenter.lbl.gov/2014/10/20/does-high-speed-network...
I heard about ESnet while consulting at Lawrence Berkeley National Laboratory.
Aspera is often used in the film industry to move large video files around. My understanding is their software takes over some layers of the network stack to make sure the pipe stays saturated. http://asperasoft.com/
Earlier quoted context omitted.
Indeed. I've been trying all sorts of weird stuff, but this takes the cake. Ubiquitous, rock-solid, sane. Plus, no worrying "is it done yet? Do I have the latest version?" Just let it run (again) - this makes it rather foolproof.
Does rsync auto-resume after failed connections?
--partial keep partially transferred filesEarlier quoted context omitted.
Amazing. If only there was away to get around the limitations at that level.
Well...two decades ago, 500 GB of data would have been moved as freight, too. Data expands to fill any available capacity - I don't think there is any way around that .
I asked why don't you just FedEx it? It's too important, the client said, and we know and trust you.
It was funny, I had it in a laptop bag, and didn't let go of it except to go through the security scanner... the only thing missing was the handcuff connecting it to my wrist. :)
Earlier quoted context omitted.
Well...two decades ago, 500 GB of data would have been moved as freight, too. Data expands to fill any available capacity - I don't think there is any way around that .
About 9 years ago, while working as a system administration consultant, I had a gig to fly a portable hard drive with about 360 GB from LA to St. Louis as part of a migration of a web application. It was faster than the network connections available to my client at the time. I remember calculating the throughput... I asked why don't you just FedEx it? It's too important, the client said, and we know and trust you. It…
Indeed, getting a trustworthy courier service is so hard that actually sending an in-house employee is worthwhile, even though their hourly rates make this extremely expensive: you are removing tens of abstraction layers, while preserving high degree of control ("Oh, we might have run it over with a truck. And accidentally put it on a plane to New Zealand on hop #3. And they can't seem to find it there.")