Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advice needed for backing up and hosting large amount of files

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 97 posts

Re: Advice needed for backing up and hosting large amount of files

#41

Post by Akronymus »

chmike wrote: post #29
Jiocus wrote: post #23

Earlier quoted context omitted.

This was the first time I heared about your issue, so I checked what they had to say on the matter. Complaints about email spam with a MEGA email address: https://help.mega.io/security/data-protection/complaints-abo... TLDR hypothesis - You get daily spam from a third party, using spoofed email headers, but I'm sure you'd figured that out already

Sorry, that is correct. I can't delete my comment. The mega.nz appears only in the from field. I see they use SPF.

You can possibly try to contact dang via hn@ycombinator.com to request that comment to be deleted. As a general rule, replied to comments can't be deleted by a user.

Re: Advice needed for backing up and hosting large amount of files

#42

Post by stavros »

rsync wrote: post #33

Please allow me to offer you a free rsync.net account, in perpetuity, for the backup portion of your requirements. We don’t do hosting of any kind so you’ll have to secure that elsewhere. Just email info@rsync.net to discuss.

Would it be against your ToS if the OP hosted a small $5 Hetzner server that mounted your FS with sshfs and hosted files with a simple proxy?

Re: Advice needed for backing up and hosting large amount of files

#43

Post by stavros »

ndtfaker wrote: post #36

Please upload to https://ipfs-gateway.cloud/ 1. This hosting is based on IPFS, which mean the data is immutable and no censorship can be apply, you can retrieve the data from another gateway in this list https://ipfs.github.io/public-gateway-checker/ . You can find out more detail about IPFS here https://ipfs.io/ 2. The owner of this hosting has donated his storage to the community and you can upload for free, this w…

IPFS is a good solution in theory, but in my experience it works so poorly in practice that I wouldn't trust it as my only form of archival. Certainly add files to IPFS too if you want, but not exclusively.

Re: Advice needed for backing up and hosting large amount of files

#45

Post by gattopalla »

stavros wrote: post #42
rsync wrote: post #33

Please allow me to offer you a free rsync.net account, in perpetuity, for the backup portion of your requirements. We don’t do hosting of any kind so you’ll have to secure that elsewhere. Just email info@rsync.net to discuss.

Would it be against your ToS if the OP hosted a small $5 Hetzner server that mounted your FS with sshfs and hosted files with a simple proxy?

Don't use hetzner for shit like this. They're EXTREMELY strict on abuse.

Re: Advice needed for backing up and hosting large amount of files

#46

Post by mootzville »

We've been using fpsync (http://www.fpart.org/fpsync/) to backup our production NAS. It is basically a scheduler wrapped around rsync. We use some 10g NICs for internal backups...it's fast enough that the SSD's are the bottleneck, so we are probably going to switch over to m.2's at somepoint.

Having said that, external transfers across the net will end up being your bottleneck, so you'll have to decide how much compression is worthwhile to reduce the time on the wire.

Another option, if available, is dump it all to drives, and snail mail it, but I'm guessing you already thought of that.

Good luck.

Re: Advice needed for backing up and hosting large amount of files

#47

Post by rsync »

stavros wrote: post #42
rsync wrote: post #33

Please allow me to offer you a free rsync.net account, in perpetuity, for the backup portion of your requirements. We don’t do hosting of any kind so you’ll have to secure that elsewhere. Just email info@rsync.net to discuss.

Would it be against your ToS if the OP hosted a small $5 Hetzner server that mounted your FS with sshfs and hosted files with a simple proxy?

No.

I will say that that sounds fragile and non performant but … at the same time, the whole point of our service is to be a “dumb” primitive - in the classic Unix sense.

So, of course this is within our terms and we see plenty of folks doing things like this.

Re: Advice needed for backing up and hosting large amount of files

#48

Post by gattopalla »

barelyusable wrote: post #6

https://www.hetzner.com/dedicated-rootserver/matrix-sx

Please don't use hetzner for things like this, atleast for the public site proxy it with another VPS, Hetzner is strict on abuse and if it's against the law of another country it will be taken down.

Re: Advice needed for backing up and hosting large amount of files

#49

Post by ChuckNorris89 »

noduerme wrote: post #14
KingZil wrote: post #4

Earlier quoted context omitted.

thanks, this is actually what we used for the initial backup, but what we foresee in the future is how to "get" the information out, because in a few decades, these media name may get forgotten all together. We also need to host the video files, which can be tricky on there.

if you want to stream the video... you need hosting in a neutral country like Switzerland and you will pay a lot for it, especially if it gets any attention at all and the unfriendly government starts to dDos it. You probably want to have it as a set of torrents, mirrors, and pop it up in different locations. Don't rely on a single locale. Again, 6Tb is not a lot. Make a site with nothing but links, hold the data in…

>you need hosting in a neutral country like Switzerland

Switzerland is only neutral on paper and has a history of letting itself and it's tech companies influenced by powerful foreign nations (the German-US - Crypto AG scandal , ProtonMail sharing info on some of its French users to the French authorities, etc.)

You might get good protection if you're a Swiss citizen living in Switzerland, but if you think Switzerland is some bastion of digital safety for foreigners, think again. If you're a foreigner, the Swiss authorities will not hesitate to throw you under the bus if you're being targeted by another powerful nation state with influence in Switzerland.

Re: Advice needed for backing up and hosting large amount of files

#50

Post by Jiocus »

bauruine wrote: post #38
Jiocus wrote: post #13

About video, MEGA.nz offers e.g. an 8TB storage tier for a reasonable price. I'm not bringing them up because of their security+privacy concept –which may suit you– but because of their take on how you can put the material to work (and do work) while hosted with them. HTML embed videos from storage into your web content and they'll be decrypted on the fly as their watched. Link sharing for selected videos for discret…

According to Kim Dotcom [0] The Chinese government has a backdoor to MEGA.nz. [0]: https://nitter.net/KimDotcom/status/1539426611870986240

If he could produce substantial evidence to back up that claim, I'd be very interested in learning more. Until then, he doesn't have the credibility to be trusted.

Based on the assumptions I have about MEGA, Chinese intelligence and their motivations I find it unlikely they have an active, mutual conspiracy (impractical, expensive).

I assume MEGA is as viable target for covert, low-cost try-your-luck attacks as much as any other western infrastructure/enterprise. In that sense and in this post-Snowden world we live in, it seems as likely that any capable nation or five-eyes member et.c. could have* such a "backdoor".

*edit: 'has' → 'could have'

Post reply on HN