I work in communication science and I know some folks who would be interested to use this for research. If you have the bandwidth, ping me at p@atrifle.net and I can pass the data on. Regarding hosting, I probably can't offer that because public institutions are typically kind of averse to hosting data for everyone.
Advice needed for backing up and hosting large amount of files
11–20 of 97 posts
Re: Advice needed for backing up and hosting large amount of files
#12consider this alternative.
Split it into parts. Create a torrent file for each part. Get a private (home) server in the US and seed it, e.g. I have 1Gb upstream and would happily donate that to it 12 hours a day. Then come back to HN and other places with a website pointing to the torrent files, asking for more people to seed it. Also, ask seeders to store mirrors and put them wherever they can. 6Tb is not that much.
Re: Advice needed for backing up and hosting large amount of files
#13About video,
MEGA.nz offers e.g. an 8TB storage tier for a reasonable price. I'm not bringing them up because of their security+privacy concept –which may suit you– but because of their take on how you can put the material to work (and do work) while hosted with them.
HTML embed videos from storage into your web content and they'll be decrypted on the fly as their watched. Link sharing for selected videos for discretionary views or downloads by receiving users, and of course team member co-op and granular access to said materials.
For starters though, prioritize getting material sent and backed up in relative safety before thinking about functionality. A simple store might fulfill your requirements in the end.
Feel free to drop me an email.
Re: Advice needed for backing up and hosting large amount of files
#14Have you looked at the Internet Archive or Archive Team? The Internet Archive provides free storage (last I knew from ~2014). https://archive.org/services/docs/api/ias3.html
thanks, this is actually what we used for the initial backup, but what we foresee in the future is how to "get" the information out, because in a few decades, these media name may get forgotten all together. We also need to host the video files, which can be tricky on there.
if you want to stream the video... you need hosting in a neutral country like Switzerland and you will pay a lot for it, especially if it gets any attention at all and the unfriendly government starts to dDos it. You probably want to have it as a set of torrents, mirrors, and pop it up in different locations. Don't rely on a single locale. Again, 6Tb is not a lot. Make a site with nothing but links, hold the data in mirrors and torrents. Also, only pay for hosting in crypto and use fake EVERYTHING. fake name fake IP, fake browser when you set it up. Let paranoia guide you. If you think you're being too paranoid, you're not being paranoid enough.
Re: Advice needed for backing up and hosting large amount of files
#15ipfs + https://www.hetzner.com/storage/storage-box
about 20€/mo, you can also setup an peertube instance to stream those video.
Re: Advice needed for backing up and hosting large amount of files
#16About video, MEGA.nz offers e.g. an 8TB storage tier for a reasonable price. I'm not bringing them up because of their security+privacy concept –which may suit you– but because of their take on how you can put the material to work (and do work) while hosted with them. HTML embed videos from storage into your web content and they'll be decrypted on the fly as their watched. Link sharing for selected videos for discret…
I get daily spam from mega.nz. They can go to hell
Re: Advice needed for backing up and hosting large amount of files
#17I'm interested in your project, have experience in this field and can give you details of privacy friendly hosting and perhaps guide you through setting this up anonymously+safely. Get a throwaway email/xmpp address and post it on site please
Thank you!
We are still in the "back up" stage, because some platforms are at risk of closing down too, now that the new chief executive is in power.
I will post a throwaway mail here if we confirm to go ahead with this plan.
Re: Advice needed for backing up and hosting large amount of files
#18I'm planning to look into https://www.storj.io/pricing (150GB free if I understand this correctly) for my own backup needs. You could theoretically host your own node somewhere on a cheap VPS, get credits and pay for your own storage? Not sure - I myself have to do some research.
Then you'd need a fronend capable mounting Storj and they seem to have pretty good documentation.
Again, unaffiliated, and not tested (yet)
Another alternative - pay google (https://one.google.com/about/plans) via a VPN in Turkey and you have 10TB for about 20USD a year. Make the files shared and there you go.
Oh and: I'm very sorry about what's happening in HK the last few years. I still hope the process is reversible.
Re: Advice needed for backing up and hosting large amount of files
#19How big of an archive are you talking about? Torrent + a few cloud providers in different regions, ipfs pinning can create sufficient redundancy?
Reddit also have data hoarders that might help
Re: Advice needed for backing up and hosting large amount of files
#20I'm not sure what to make of the monero mention, is it a requirement that you can pay anonymously? or do you have a fund that is able wire money? If you're avoiding the eye of China I suppose it's the former. I believe there's services to "pin" content to the IPFS network, but I'd be surprised if there aren't torrent seedboxes you can pay for with crypto, but I'm afraid I'm short on specifics. Would the idea be, to k…
Monero would allow donations to be anonymous, so even if all things go south, it should be only my payment to VPS/hosting that would be leaked.
I want to enforce this by only accepting monero, having privacy by default, at the cost of not getting a lot of donations.