Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advice needed for backing up and hosting large amount of files

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 97 posts

Re: Advice needed for backing up and hosting large amount of files

#31

Post by uniqueuid »

null0pointer wrote: post #26
uniqueuid wrote: post #11

I work in communication science and I know some folks who would be interested to use this for research. If you have the bandwidth, ping me at p@atrifle.net and I can pass the data on. Regarding hosting, I probably can't offer that because public institutions are typically kind of averse to hosting data for everyone.

While I trust that the commenter above is genuine, OP seems to be highly concerned about their opsec. So I just want to post a friendly reminder for the OP to be careful about reaching out to people that comment on this post. For example, don’t reach out using an email that can be linked to your identity.

Yes I thought about that. There are a couple of different threat models. Anyone living in HK right now is in great danger and should be extremely cautious. For them, even posting here may be too risky.

Anyone outside of China may still be at risk, especially if they have family / friends there.

But to add a bit of opsec, the email above is on Protonmail, and if OP wants to reach out they can create a burner account there and email me - that would stay within the service. (cue debate about TLA access to Protonmail, but at least it's probably not the CCP)

Re: Advice needed for backing up and hosting large amount of files

#34

Post by smcleod »

rsync wrote: post #33

Please allow me to offer you a free rsync.net account, in perpetuity, for the backup portion of your requirements. We don’t do hosting of any kind so you’ll have to secure that elsewhere. Just email info@rsync.net to discuss.

It's lovely to see things like this.

Re: Advice needed for backing up and hosting large amount of files

#35

Post by zxcvbn4038 »

Keep a copy in AWS S3, they are extremely reliable for storage, you’ll sleep well knowing they are there. But that S3 bucket should be private and locked down - it’s your master copy - and AWS is the most expensive option for outgoing transit so you want to use it as little as possible. For routine usage keep a second copy in Backblaze B2 which is the cheapest storage you can get that isn’t running out of someone’s basement. I’d use Digital Ocean to serve files from that - doesn’t have to be anything major, just something running nginx. I’d front that with Cloudflare which is essentially free for content that doesn’t change frequently. Your DO instance should only respond to requests from Cloudflare’s backend IPs and only if those requests contain some magic header you inject. That makes it near impossible to find your DO host and access it directly, and it is the only thing that knows your backblaze B2 secrets so nobody is accessing B2 directly either.

If you want to add an obfuscation layer in front of Cloudflare - though maybe not needed in your case since the content itself isn’t illegal in most of the world - but if you want to serve through Tor to protect Cloudflare you can - DO is a good option again - and if you go that route your Cloudflare site should have some random unrelated name and should only serve files if the requests have a magic header. Tor is where your going to reverse all the money you save with the Cloudflare + DO + B2 setup, your going to pay mostly for network usage, Tor has a lot of overhead. You can scale Tor horizontally across multiple hidden services if you can afford it.

Re: Advice needed for backing up and hosting large amount of files

#36

Post by ndtfaker »

Please upload to https://ipfs-gateway.cloud/

1. This hosting is based on IPFS, which mean the data is immutable and no censorship can be apply, you can retrieve the data from another gateway in this list https://ipfs.github.io/public-gateway-checker/. You can find out more detail about IPFS here https://ipfs.io/

2. The owner of this hosting has donated his storage to the community and you can upload for free, this was original posted on Reddit at: https://www.reddit.com/r/ipfs/comments/v0bnd1/ipfsgatewayclo.... He has approximately 13.7PB of storage =)))

After upload, you might want to save all the CID of your data and share them with your people (just like a URL)

Re: Advice needed for backing up and hosting large amount of files

#37

Post by laurent123456 »

zxcvbn4038 wrote: post #35

Keep a copy in AWS S3, they are extremely reliable for storage, you’ll sleep well knowing they are there. But that S3 bucket should be private and locked down - it’s your master copy - and AWS is the most expensive option for outgoing transit so you want to use it as little as possible. For routine usage keep a second copy in Backblaze B2 which is the cheapest storage you can get that isn’t running out of someone’s b…

If using S3 don't forget to setup a lifecycle policy to transition to a cheaper storage like S3 Intelligent Tiering, and possibly Glacier. Maybe just avoid One-Zone IA for better durability.

Re: Advice needed for backing up and hosting large amount of files

#38

Post by bauruine »

Jiocus wrote: post #13

About video, MEGA.nz offers e.g. an 8TB storage tier for a reasonable price. I'm not bringing them up because of their security+privacy concept –which may suit you– but because of their take on how you can put the material to work (and do work) while hosted with them. HTML embed videos from storage into your web content and they'll be decrypted on the fly as their watched. Link sharing for selected videos for discret…

According to Kim Dotcom [0] The Chinese government has a backdoor to MEGA.nz.

[0]: https://nitter.net/KimDotcom/status/1539426611870986240

Re: Advice needed for backing up and hosting large amount of files

#39

Post by pronoiac »

KingZil wrote: post #20
jazzyjackson wrote: post #5

I'm not sure what to make of the monero mention, is it a requirement that you can pay anonymously? or do you have a fund that is able wire money? If you're avoiding the eye of China I suppose it's the former. I believe there's services to "pin" content to the IPFS network, but I'd be surprised if there aren't torrent seedboxes you can pay for with crypto, but I'm afraid I'm short on specifics. Would the idea be, to k…

Monero would allow donations to be anonymous, so even if all things go south, it should be only my payment to VPS/hosting that would be leaked. I want to enforce this by only accepting monero, having privacy by default, at the cost of not getting a lot of donations.

Torrent seedboxes might also be useful, as they prioritize bandwidth and (maybe?) disk space. I'm not sure if they'd be more or less secure than the typical VPS hosting, though, and I'd default to pessimism.

Re: Advice needed for backing up and hosting large amount of files

#40

Post by Akronymus »

bauruine wrote: post #38
Jiocus wrote: post #13

About video, MEGA.nz offers e.g. an 8TB storage tier for a reasonable price. I'm not bringing them up because of their security+privacy concept –which may suit you– but because of their take on how you can put the material to work (and do work) while hosted with them. HTML embed videos from storage into your web content and they'll be decrypted on the fly as their watched. Link sharing for selected videos for discret…

According to Kim Dotcom [0] The Chinese government has a backdoor to MEGA.nz. [0]: https://nitter.net/KimDotcom/status/1539426611870986240

Huh. I just learned that nitter, but not twitter is blocked by the company firewall.

But yeah, I don't trust mega either.

Post reply on HN