Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I store my files

unixsheikh.com

Post reply on HN
Print view View original

111–120 of 144 posts

Re: How I store my files

#111

Post by tedcrilly »

I'm in the self-hosted camp as well with two Synology NAS boxes in two different countries that all of the family's computers synchronize to using Syncthing. Each of the boxes runs a local backup to an external drive nightly and one of them also runs a backup of a subset of (really important) folders to Backblaze. This uses Synology's proprietary backup tool (HyperBackup) which I intend to replace with an open source solution (most likely rclone but I'd be interested in suggestions). As an additional measure I rotate the external drives in one location weekly. So far it proved quite reliable when switching machines (in combination with git-managed dotfiles and stow) and accessing data on demand. I also make a full image of my laptop on an external drive more or less once a month to enable quick restore in case my OS gets hosed. One problem I still only semi-solved is synchronizing iPhones. Right now we just synchronize photos using Synology's DS File and I do an iTunes backup using a Windows VM which is clunky. High hopes for libimobiledevice here but I've had no time to properly research it yet.

All of the above requires some work but it's fun more than a nuisance. Probably not that great of a solution when I'm no longer around though.

Re: How I store my files

#112

Post by gurchik »

kumarsw wrote: post #106
gurchik wrote: post #93

Earlier quoted context omitted.

> I'll agree with the author partly on software such as rustic/duplicacy - they are not a good solution for long-term archiving, nor are they marketed as such. I self host a Nextcloud server and I run duplicity each night to back this up to an offsite location. I use incremental backups because I have a data cap. From there each month I duplicate the most recent versions of the Duplicity directory onto a cold hard dr…

My understanding is that there are two issues. First, modern deduplicating backup software like borg/restic/duplicacy store data in a repository in unique chunks. This avoids the issue that incremental backup software like duplicity have where they can create long chains of incremental changes which is slow to restore and increases the likelihood of errors on restore. Second, both deduplicating and incremental backup…

Got it. So perhaps after the Duplicity files have been incrementally uploaded to the remote datacenter, for the cold storage backup of those files rather than simply duplicating the Duplicity files I should unpack them and then rearchive them into a single flat encrypted archive.

Re: How I store my files

#113

Post by jarenmf »

unilynx wrote: post #101
jarenmf wrote: post #5

I use borg backup to sync all our files (from ~5 machines) to an old PC which synchronizes everything to the cloud. Borg has deduplication, compression, encryption, and saves several versions of the files which I think is crucial. Also all text-based data are on Git like software, papers, etc.

Is your PC synchronising the source files or the borg files to the cloud ? Otherwise it's still a SPOF for your backups (the borg files might corrupt, and you might be syncing that corruption to the cloud)

Good point. I am indeed synchronizing the borg file. I have however my script run borg check before it synchronizes to the cloud. It would be better to synchronize the source files but the encryption and deduplication are what's keeping me from doing that.

Re: How I store my files

#114

Post by tailspin2019 »

pricecomstock wrote: post #72
visviva wrote: post #11

Darn, I was hoping this would be an article about organizing one's files. I was really in the mood for reading about that, then spending the rest of the morning reworking my own system.

I have used this system in certain areas before. It's decent. But it's interesting to read about if you're in the mood! https://johnnydecimal.com/

Oh man, that's a cool approach.

Down the rabbit hole I go...

Re: How I store my files

#115

Post by mopsi »

TchoBeer wrote: post #71
bachmeier wrote: post #66

> Everything can look really nice "on paper" but you don't know what goes on behind the scenes. I have worked with a lot of different people and I have seen too much crazy shit to fully trust anyone with my important data. A cloud provider may have the best of intentions, but sometimes all it takes is a single grumpy employee or even a minor mistake to do a lot of damage. OneDrive and Google Drive are both pretty che…

Alternatively, "sure all my sensitive data was stolen and leaked, but at least I made it to Italy" said no one ever.

Data stored in Google Drive etc should be encrypted and split into 50 MB chunks or something like that to hide metadata and mitigate the risk of leaks. Better backup tools have been offering this for a long time.

Re: How I store my files

#116

Post by onefuncman »

tbronchain wrote: post #104
yosito wrote: post #96

I'm a digital nomad, and I decided I wanted to store all of my data as securely as possible without relying on Google/Amazon/Microsoft/Apple. Here's my setup, in case anyone finds it useful. I store all of my data in Nextcloud hosted on a VPS. It's virtually impossible to guarantee the security of data on a running VPS, so my sensitive data is also encrypted with Cryptomator, so I don't have to trust my VPS host (but…

Wondering something, if you are using cryptomator and a VPS - so you don't own your data and use a higher level encryption - why not using the cloud storage services (i.e Dropbox, icloud etc.) ?

I put my 1Password vault on Dropbox specifically because I can install both apps via iOS store on a new phone and bootstrap my setup with just 2 passwords.

iCloud is actually too hard to recover in case of hardware loss, I don't recommend it to anyone.

Re: How I store my files

#117

Post by aborsy »

gurchik wrote: post #112
kumarsw wrote: post #106

Earlier quoted context omitted.

My understanding is that there are two issues. First, modern deduplicating backup software like borg/restic/duplicacy store data in a repository in unique chunks. This avoids the issue that incremental backup software like duplicity have where they can create long chains of incremental changes which is slow to restore and increases the likelihood of errors on restore. Second, both deduplicating and incremental backup…

Got it. So perhaps after the Duplicity files have been incrementally uploaded to the remote datacenter, for the cold storage backup of those files rather than simply duplicating the Duplicity files I should unpack them and then rearchive them into a single flat encrypted archive.

>>I should unpack them and then rearchive them into a single flat encrypted archive.

Which is a duplicity full back up (that one would normally do once a month or so).

Deduplication is an additional potential point of failure.

Re: How I store my files

#118

Post by hunter-gatherer »

yepthatsreality wrote: post #108

> Not only does encryption during data recovery make everything much more difficult, but should you pass away, your family members might not have the skills required to access the data. Terrible advice regarding encryption. Really if you have important data that your family needs at the time of your death you should have a plan for that as well. Not avoid encryption because it’s “too hard”.

Exactly. Encrypt with a simple utility, and write down the password and store in a safe place and let someone know. In my case, I have a friend who is very technical and I can trust to assist my spouse (who is not technical) with some items if need be. Part of my hand-written instructions are his phone number/email, and to call him if there are issues decrypting or restoring data.

Re: How I store my files

#119

Post by soupbowl »

dangerface wrote: post #60

> These tools work really great, until they don't! Huh? Whats the problem? How is ZFS any different because its lower level? > ZFS without ECC memory is no worse than any other file system without ECC memory. You really do need ECC memory unless you are ok with your pool becoming corrupt every 6 months or so. Im just saying this from my experience of running a ZFS server without ECC, the data wasn't critical so I lef…

I stored 2tb of actively used data on zfs without ecc ram from freebsd 8 to 12. I had no noticeable corruption. I only post this because there is a weird assumption that not using ecc ram is a death sentence for your zfs data.

I use ecc ram now and I think it's the proper way to do zfs, but let's not pretend you forfeit your data by not using ecc.

Re: How I store my files

#120

Post by kureikain »

bachmeier wrote: post #66

> Everything can look really nice "on paper" but you don't know what goes on behind the scenes. I have worked with a lot of different people and I have seen too much crazy shit to fully trust anyone with my important data. A cloud provider may have the best of intentions, but sometimes all it takes is a single grumpy employee or even a minor mistake to do a lot of damage. OneDrive and Google Drive are both pretty che…

Time to time I read about people who are randomly banned/locked account by google. I had 1.5TB of my precious memory of my daughter all the way back when she was born,

But I keep a local backup on my old mac, basically a bunch of external hard-drive but it's a pain in the ass to manage thse.

Post reply on HN