Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advice needed for backing up and hosting large amount of files

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 97 posts

Re: Advice needed for backing up and hosting large amount of files

#21

Post by ilaksh »

Internet archive and IPFS pinning on Filebase and web3.storage as well as your own IPFS cluster put together with volunteers. Also can look into Sia and Filecoin.

If you can get two other engineers to volunteer and about 5 or 6 volunteers for buying storage, you can cover all of the above in less than a month.

Easiest is probably Internet Archive though. With a torrent.

Look into CAR files. Once you get those it's easy to pin on IPFS with multiple providers. Some of them free or extremely cheap.

Re: Advice needed for backing up and hosting large amount of files

#22

Post by [deleted] »

noduerme wrote: post #14
KingZil wrote: post #4

Earlier quoted context omitted.

thanks, this is actually what we used for the initial backup, but what we foresee in the future is how to "get" the information out, because in a few decades, these media name may get forgotten all together. We also need to host the video files, which can be tricky on there.

if you want to stream the video... you need hosting in a neutral country like Switzerland and you will pay a lot for it, especially if it gets any attention at all and the unfriendly government starts to dDos it. You probably want to have it as a set of torrents, mirrors, and pop it up in different locations. Don't rely on a single locale. Again, 6Tb is not a lot. Make a site with nothing but links, hold the data in…

[deleted]

Re: Advice needed for backing up and hosting large amount of files

#23

Post by Jiocus »

chmike wrote: post #16
Jiocus wrote: post #13

About video, MEGA.nz offers e.g. an 8TB storage tier for a reasonable price. I'm not bringing them up because of their security+privacy concept –which may suit you– but because of their take on how you can put the material to work (and do work) while hosted with them. HTML embed videos from storage into your web content and they'll be decrypted on the fly as their watched. Link sharing for selected videos for discret…

I get daily spam from mega.nz. They can go to hell

This was the first time I heared about your issue, so I checked what they had to say on the matter.

Complaints about email spam with a MEGA email address: https://help.mega.io/security/data-protection/complaints-abo...

TLDR hypothesis - You get daily spam from a third party, using spoofed email headers, but I'm sure you'd figured that out already

Re: Advice needed for backing up and hosting large amount of files

#25

Post by softwareoiu »

chmike wrote: post #16
Jiocus wrote: post #13

About video, MEGA.nz offers e.g. an 8TB storage tier for a reasonable price. I'm not bringing them up because of their security+privacy concept –which may suit you– but because of their take on how you can put the material to work (and do work) while hosted with them. HTML embed videos from storage into your web content and they'll be decrypted on the fly as their watched. Link sharing for selected videos for discret…

I get daily spam from mega.nz. They can go to hell

No post body was provided.

Re: Advice needed for backing up and hosting large amount of files

#26

Post by null0pointer »

uniqueuid wrote: post #11

I work in communication science and I know some folks who would be interested to use this for research. If you have the bandwidth, ping me at p@atrifle.net and I can pass the data on. Regarding hosting, I probably can't offer that because public institutions are typically kind of averse to hosting data for everyone.

While I trust that the commenter above is genuine, OP seems to be highly concerned about their opsec. So I just want to post a friendly reminder for the OP to be careful about reaching out to people that comment on this post. For example, don’t reach out using an email that can be linked to your identity.

Re: Advice needed for backing up and hosting large amount of files

#28

Post by herbst »

noduerme wrote: post #14
KingZil wrote: post #4

Earlier quoted context omitted.

thanks, this is actually what we used for the initial backup, but what we foresee in the future is how to "get" the information out, because in a few decades, these media name may get forgotten all together. We also need to host the video files, which can be tricky on there.

if you want to stream the video... you need hosting in a neutral country like Switzerland and you will pay a lot for it, especially if it gets any attention at all and the unfriendly government starts to dDos it. You probably want to have it as a set of torrents, mirrors, and pop it up in different locations. Don't rely on a single locale. Again, 6Tb is not a lot. Make a site with nothing but links, hold the data in…

> you need hosting in a neutral country like Switzerland and you will pay a lot for it

If you are actually interested in Swiss hosting for whatever reason, there are plenty of VPS & Dedicated offers these days, like Exoscale, that offer really fair pricing comparable to Vultr or DO

Re: Advice needed for backing up and hosting large amount of files

#29

Post by chmike »

Jiocus wrote: post #23
chmike wrote: post #16

Earlier quoted context omitted.

I get daily spam from mega.nz. They can go to hell

This was the first time I heared about your issue, so I checked what they had to say on the matter. Complaints about email spam with a MEGA email address: https://help.mega.io/security/data-protection/complaints-abo... TLDR hypothesis - You get daily spam from a third party, using spoofed email headers, but I'm sure you'd figured that out already

Sorry, that is correct. I can't delete my comment. The mega.nz appears only in the from field. I see they use SPF.
Post reply on HN