Advice needed for backing up and hosting large amount of files • Hacker News Board

Re: Advice needed for backing up and hosting large amount of files

#81

Post by kkielhofner » Wed, Jul 06, 2022, 1:58 PM UTC

stavros wrote: post #43

ndtfaker wrote: post #36

Please upload to https://ipfs-gateway.cloud/ 1. This hosting is based on IPFS, which mean the data is immutable and no censorship can be apply, you can retrieve the data from another gateway in this list https://ipfs.github.io/public-gateway-checker/ . You can find out more detail about IPFS here https://ipfs.io/ 2. The owner of this hosting has donated his storage to the community and you can upload for free, this w…

IPFS is a good solution in theory, but in my experience it works so poorly in practice that I wouldn't trust it as my only form of archival. Certainly add files to IPFS too if you want, but not exclusively .

I second this and for anything other than "need to fetch a file or two occasionally" it's essentially broken in my mind. Expect to wait anywhere from 3 - 30 seconds for first byte. When modern CDNs are doing first byte in 100ms (or whatever) over HTTPS that's an eternity for most users.

Virtually all IPFS storage and bandwidth from pinning and gateway providers is at least 2-3x what you'd expect to pay for S3, etc (because they just use S3 on the backend). The public IPFS gateways have such low request limits and especially poor performance they often struggle to load a website with more than a couple of IPFS hosted assets on it.

If you want to run your own node go-ipfs is extremely difficult if not impossible to use at anything more than toy scale. Eats RAM like crazy, uses a TON of bandwidth (not unexpected but still seems like a lot), garbage collection is broken, and much more...

Frankly the state of IPFS is embarrassing considering it's seven year old tech.

Re: Advice needed for backing up and hosting large amount of files

#82

Post by cbsks » Wed, Jul 06, 2022, 2:08 PM UTC

rsync wrote: post #33

Please allow me to offer you a free rsync.net account, in perpetuity, for the backup portion of your requirements. We don’t do hosting of any kind so you’ll have to secure that elsewhere. Just email info@rsync.net to discuss.

I just want to say, mysterious ’rsync’ person, your evangelism of your company is top notch. From this and other comments you have made on this site, rsync.net is a service that I will sign up for without hesitation the instant I have offsite backup needs.

Re: Advice needed for backing up and hosting large amount of files

#83

Post by fsflover » Wed, Jul 06, 2022, 3:36 PM UTC

ajvs wrote: post #56

fsflover wrote: post #30

I wonder why nobody mentioned torrents over I2P: https://geti2p.net . It's distributed, anonymous and anyone can help without risking to reveal their IP. It's slow of course but might be faster if many people join.

What torrent clients offer native I2P support?

HN audience should be able to install the I2P client and enter the address in browser to access the torrents.

Re: Advice needed for backing up and hosting large amount of files

#84

Post by solardev » Wed, Jul 06, 2022, 3:53 PM UTC

janandonly wrote: post #80

I am surprised that between all these technically minded people, no one suggested making a BitTorrent seeder box and a tracker?

I think that works well for popular torrents (Wikileaks, etc.) but not so well for things that only a few people would download. The seedbox ends up being the primary host anyway, which is kinda what the OP planned already (i.e., a torrent yes, but in conjunction with offsite backup + http hosting).

Bittorrent isn't a guarantee of reliability, it's just a way to distribute files among multiple seeders. But that assumes there will be many seeders, which for something like this probably isn't the case.

Re: Advice needed for backing up and hosting large amount of files

#85

Post by solardev » Wed, Jul 06, 2022, 3:59 PM UTC

6figurelenins wrote: post #69

> Some of you may know Hong Kong has been attacked by China and most of the pro democratic media companies have shut down... I also would like to host these files on a public domain, just to ensure more people can revisit on these events in the future. I'm interested now. What happened?

I think it was a while ago. During COVID and right before, China started moving in on Hong Kong, clamping down on dissent and pro-democracy movements, shutting down protests and arresting individuals involved with media companies. It was the end of the Hong Kong free press.

https://en.wikipedia.org/wiki/Apple_Daily

https://en.wikipedia.org/wiki/Hong_Kong_national_security_la...

https://en.wikipedia.org/wiki/Internet_censorship_in_Hong_Ko...

Re: Advice needed for backing up and hosting large amount of files

#86

Post by janandonly » Wed, Jul 06, 2022, 5:04 PM UTC

fsflover wrote: post #30

I wonder why nobody mentioned torrents over I2P: https://geti2p.net . It's distributed, anonymous and anyone can help without risking to reveal their IP. It's slow of course but might be faster if many people join.

The real issue is not making a torrent file, but finding enough people who want to seed it for years to come, whether over clearnet, TOR or I2P.

Re: Advice needed for backing up and hosting large amount of files

#87

Post by ev1 » Wed, Jul 06, 2022, 5:20 PM UTC

elcritch wrote: post #61

gattopalla wrote: post #45

Earlier quoted context omitted.

Don't use hetzner for shit like this. They're EXTREMELY strict on abuse.

How is this abuse? You're paying hetzner for a VPS. If running a proxy server is an abuse of their TOS, then in my mind their service doesn't seem much more than a scam. I'm glad to know to avoid their service.

If this is HK/CN related stuff, I'd anticipate Hetzner to be drowned in torrential rains of fake abuse emails.

Re: Advice needed for backing up and hosting large amount of files

#88

Post by indrora » Wed, Jul 06, 2022, 7:34 PM UTC

Surprised nobody's suggested this:

10TB HDDs are cheap now. Three of them, with either lvm crypto or some other encryption system.

Three people leave the country on three different airlines to three different countries at three different times.

Then you sort out hosting the data.

Re: Advice needed for backing up and hosting large amount of files

#89

Post by RealStickman_ » Wed, Jul 06, 2022, 7:40 PM UTC

xrisk wrote: post #63

stavros wrote: post #42

Earlier quoted context omitted.

Would it be against your ToS if the OP hosted a small $5 Hetzner server that mounted your FS with sshfs and hosted files with a simple proxy?

Whoa that’s a cool trick! It’s important to note, however, that sshfs has been archived by its creator. What this entails with regard to the future of the project is uncertain though.

I'm not sure what protocols rsync.net exposes, but it's probably supported by rclone if there are others than ssh.

Re: Advice needed for backing up and hosting large amount of files

#90

Post by sophacles » Wed, Jul 06, 2022, 8:37 PM UTC

noduerme wrote: post #12

consider this alternative. Split it into parts. Create a torrent file for each part. Get a private (home) server in the US and seed it, e.g. I have 1Gb upstream and would happily donate that to it 12 hours a day. Then come back to HN and other places with a website pointing to the torrent files, asking for more people to seed it. Also, ask seeders to store mirrors and put them wherever they can. 6Tb is not that much.

If you do a plan like this OP, also share the torrent links and explanation to r/datahoarder - those folks are always talking about interesting datasets that they think are useful to seed. Many of them explicitly do so to theoretically fight against the sort of censorship you are actually facing, and would jump at the opportunity to help.