Live data from Hacker News

Ask HN: What is your favorite method of sending large files?

news.ycombinator.com

121–130 of 363 posts

Re: Ask HN: What is your favorite method of sending large files?

#121
post #86

Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice. Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea eith…

> Unlike HTTP, you get reasonable retry behavior on network hiccups

Seriously. Zmodem/resume was doing better in 1986.

Re: Ask HN: What is your favorite method of sending large files?

#122
post #86

Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice. Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea eith…

Can you recommend easy software to do the creation? Don't you need a tracker as well?

transmission is pretty complete and easy to use

Re: Ask HN: What is your favorite method of sending large files?

#123

Earlier quoted context omitted.

piping through tar is my "power move" when I want to steer a new recruit. I just do it casually in front of them... depending on whether they are awed or horrified by that, they'll get more hacky or more formal work.

You can also pipe through gpg if you want encryption along the way!

And through gzip/zstd/etc if you want compression!

Re: Ask HN: What is your favorite method of sending large files?

#124

5GB on my local network: Windows File Share, Rsync, or HTTP depending on source/destination 5GB on Internet: Upload to my OVH VPS and HTTP or Rsync it to its destination 10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! https://www.tidbitsfortechs.com/2013/09/never-underestimate-... And since you now buy 1TB micro SD cards, so perhaps I'd split th…

OVH is great if it’s not business critical! As the semi-official safe haven of piracy in Europe, OVH’s network has uhh...seen better days.

Re: Ask HN: What is your favorite method of sending large files?

#125

Netcat: $ nc -l 4242 > dest And then on the sending end: $ nc hostname 4242 This works great when you just need to get a file of any size from one machine to another and you’re both on the same network. Are used this a lot at one of my offices to schlep the files around between a few of the machines we had.

and if you are not worried about corruption. You might at least want to run a checksum on both ends and compare them.

Nah, unix nerds don't care about data corruption. That's the responsibility of some other layer, so by definition we're not in trouble if the whole thing blows up by our own negligence.

Checkmate, rsync.

Re: Ask HN: What is your favorite method of sending large files?

#127
post #86

Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice. Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea eith…

Sometimes I wonder if the bad connotation due to piracy made this protocol not standard (in a sense, that's not widely adopted, and people need dedicated clients, instead of it working for example in browsers). I don't know enough, maybe it has security issues, or some limits... but the fact is that it's still widely used, by tech literate people, so it still stands the test of time and it must be doing something rig…

> people need dedicated clients, instead of it working for example in browsers

"Version 9.0 [...], released on June 20, 2006" "Opera introduced [...] a built-in BitTorrent client" [1]

That's history now, of course, courtesy of today's browser monoculture.

[1] https://en.wikipedia.org/wiki/History_of_the_Opera_web_brows...

Re: Ask HN: What is your favorite method of sending large files?

#128
Not something I've used, and it's difficult to countenance anything called that, but what Globus (globus.org) has become seems to be quite popular for transferring large research datasets between academic sites which subscribe.

For ssh, there's https://www.psc.edu/index.php/hpn-ssh for improving wide-area transfer speeds, and something else, I think.

Post reply on HN