Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your favorite method of sending large files?

news.ycombinator.com

Post reply on HN
Print view View original

231–240 of 363 posts

Re: Ask HN: What is your favorite method of sending large files?

#231

Post by Matthias247 »

gpm wrote: post #169
Matthias247 wrote: post #75

Earlier quoted context omitted.

and if you are not worried about corruption. You might at least want to run a checksum on both ends and compare them.

Netcat is tcp is "reliable" and has checksums for each packet, as long as the length is right you can be reasonably assured that the file transferred correctly. Not that extra checksums are a bad idea...

TCP indeed has checksums! But they are known not to offer ultimate protection. I had to deal with hosts corrupting TCP streams due to bad hardware before. Which was luckily detected by higher level checksums.

Re: Ask HN: What is your favorite method of sending large files?

#232

Post by brippalcharrid »

geocrasher wrote: post #57

5GB on my local network: Windows File Share, Rsync, or HTTP depending on source/destination 5GB on Internet: Upload to my OVH VPS and HTTP or Rsync it to its destination 10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! https://www.tidbitsfortechs.com/2013/09/never-underestimate-... And since you now buy 1TB micro SD cards, so perhaps I'd split th…

You wouldn't want to keep all of your proverbial eggs in one basket. Split the file into a multi-volume archive and create a par2/par3 parity volume set. Distribute parts of the parity set and parts of the data archive amongst m pigeons such that you only need n pigeons to arrive on time without having gotten lost, shot or eviscerated by hawks, and you can reliably rebuild the original data set with fault tolerance that also handles read/write errors on the MicroSD cards.

Re: Ask HN: What is your favorite method of sending large files?

#234

Post by chid »

voltagex_ wrote: post #209
arsome wrote: post #179

Earlier quoted context omitted.

Just be careful - unless you mark that torrent private it'll get posted to the DHT and crawlers like BtDig will pick it up and list it publicly. For this reason I prefer using something like Syncthing which is designed more with this purpose in mind.

How are BtDig crawling the DHT? I thought this was made impossible or at least more difficult a while ago.

I don't see how it was made impossible. It's definitely possible, you can do it yourself there's many examples on github (even btdig has an old version on it)

Re: Ask HN: What is your favorite method of sending large files?

#235

Post by voltagex_ »

terramex wrote: post #131
geocrasher wrote: post #57

5GB on my local network: Windows File Share, Rsync, or HTTP depending on source/destination 5GB on Internet: Upload to my OVH VPS and HTTP or Rsync it to its destination 10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! https://www.tidbitsfortechs.com/2013/09/never-underestimate-... And since you now buy 1TB micro SD cards, so perhaps I'd split th…

10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! In Poland there is an option to send package via train for 8$. Just go to station and give package to conductor/train guard, few hours later receiver can get it at destination station. Sending HDDs full of raw video footage this way is very popular among video editors here.

Oh that's fantastic - do you know how many packages they take? How does the receiver know when their package is coming? (other than the train timetable)

Re: Ask HN: What is your favorite method of sending large files?

#236

Post by efreak »

livueta wrote: post #86

Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice. Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea eith…

I had to transfer 2tb of data over a wireless connection that kept dropping packets. BitTorrent was the only method I found that worked; everything else either broke down when it disconnected, or had too low of a retry limit to be useful, or wouldn't resume properly.

Re: Ask HN: What is your favorite method of sending large files?

#237

Post by a1371 »

qrv3w wrote: post #222
spurgu wrote: post #186

Earlier quoted context omitted.

I don't see an option to send text (browsed through README and Issues), is this not possible with croc? With Magic Wormhole it's the default to get a message prompt when running "wormhole send". Or you can do it as a oneliner with "wormhole send --text 'something'". Wormhole William can do this as well IIRC. I use MWH often for sending stuff like URL's, passwords, API keys... having to create a file for that is quite…

Just added it now. Send with croc send --text "hello, world" [1] [1]: https://github.com/schollz/croc/releases/tag/v8.2.0

Haha, quick turnaround.

Re: Ask HN: What is your favorite method of sending large files?

#238

Post by bigiain »

simonebrunozzi wrote: post #165
geocrasher wrote: post #57

5GB on my local network: Windows File Share, Rsync, or HTTP depending on source/destination 5GB on Internet: Upload to my OVH VPS and HTTP or Rsync it to its destination 10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! https://www.tidbitsfortechs.com/2013/09/never-underestimate-... And since you now buy 1TB micro SD cards, so perhaps I'd split th…

> Never underestimate the bandwidth of a station wagon full of backup tapes! The exact quote [0] by Andrew Tanenbaum is: "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.". Obligatory xkcd [1]. I met Andy Tanenbaum in 2008 at Vrije Universiteit in Amsterdam. I was the Technology Evangelist for AWS in Europe, giving a seminar there on cloud computing. I loved him. Funny, sm…

Some 2010 vintage calculations, when 16GB microSD cards were considered "big", estimates 275PB would fit in a 1985 Volvo 240:

http://dansdata.com/gz105.htm

A 2020 update for that would probably use 256 or 512GB cards resulting in around 4.4 or 8.8EB (And probably twice that again if the cargo capacity of a typical 2020 SUV equivalent of a 1985 Volvo is assumed...)

Re: Ask HN: What is your favorite method of sending large files?

#239

Post by Brian-Puccio »

chillfox wrote: post #211
godelski wrote: post #197

Earlier quoted context omitted.

I've sent flash media through mail without problems. As well as hard drives with disks. Never had an issue.

I would assume that mail scanning is similar to the x-ray scanner at the airport, and any SD card or hard drive that cant handle that is useless. Could you imagine having your laptop and camera wiped every time you jumped on a plane.

Film is mailed all of the time without issue. Film would be ruined by the luggage scanning (i.e. higher power) machines though most film is usually OK with the lower power carry-on scanning machines.

Re: Ask HN: What is your favorite method of sending large files?

#240

Post by talove »

- Airdrop for immediate stuff. Gets dicey around 30-50GB tho.

- iCloud drive for personal stuff I need to share between devices. I trust this to sync anything up to a TB reliably.

- Google Drive when I need to share to someone else.

- Occasionally external drives when I need to move data fast locally.

- Some combination of S3 / AWS CLI / EC2 when things go beyond personal computer capacity depending on where the data is coming from and going to.

Post reply on HN