Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you manage your “family data warehouse”?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 141 posts

Re: Ask HN: How do you manage your “family data warehouse”?

#121

Post by Helmut10001 »

coderatlarge wrote: post #103
Helmut10001 wrote: post #53

I separate my data into two categories, fast (and small) and slow (but big). Both types are important, but separation makes management much easier. Fast data: - documents, pdfs, config files, notes txt etc. - these files are synced directly, on-site and off-site Slow data: - archives, images, music, large backups etc. - these files are transfered to off-site archive on a per-need basis - they are made available in th…

if you're willing share: very roughly, how long did it take to setup and how long does it take to run incrementally?

I (deliberately) decided in 2017 to make myself (and my family) independent from public cloud providers and SAS. The initial part was quite a bit of work, e.g. Proxmox hypervisor hardware rack setup etc, but I also learned a lot and I must say, every step felt like a tiny relief of independence.

Since about 2-3 years, it is a more gradual progression. I only need about 1-2 hours monthly for administration work, since I automated many of these things:

- Nextcloud (like most of my services) is setup in Docker/compose and updated using Watchtower, except for major updates (e.g. 25.x.x to 26.x.x), where I want to be present

- I update my VMs (LXC) using Ansible; it is a single command that is started daily on my work laptop that updates all 12 VMs

- Backups are automated for the "fast" data category: A maintenance VM starts up my remote offsite backup though IPSEC, unlocks LUKS volume on a NAS and starts the Borgmatic job; after it is done, the LUKS is locked and the NAS shutdown (I plan to replace this with ZFS, too, which will further simplify things)

- The slow data (rync) backup is still done manually, I login every 1-2 month and copy & paste the command. I could automate this, too; but I want to be present here. Also, since ZFS and automated snapshots, I don't have as much need for a higher frequency of backups anymore.

- Once a year, all my data is backed up to external drives that I need to manually connect and then do a `zfs send`, to update snapshots.

- phones, laptops etc. of the whole family sync their content (photos, contacts, files) to Nextcloud; phone stuff is automatically removed locally from devices, once synced to the server; from there they're archived and backed up

If you ask how long the actual backup jobs require:

- I have only slow bandwidth to my offsite backup (5000kbit)

- borgmatic is still very fast, about 2-3 hours weekly

- slow data (rsync) usually takes 1-2 days every month (about 80GB on average change here; for about 5 family members)

Re: Ask HN: How do you manage your “family data warehouse”?

#122

Post by vr46 »

rufus_foreman wrote: post #75
vr46 wrote: post #24

A Synology NAS running Portainer ( https://www.portainer.io/ ) running Paperless NGX ( https://github.com/paperless-ngx/paperless-ngx ) This works better than I can possibly tell you. I have an Epson WorkForce ES-580W that I bought when my mother passed away to bulk scan documents and it scans everything, double-sided if required, multi-page PDFs if required, at very high speed and uploads everything to OneDrive, at…

I haven't looked at Portainer before, what does it give you that just running a Docker instance on the NAS doesn't do?

Well, a lovely management system, basically. One can run and edit Docker Compose configurations in the browser, see everything that’s running, manage environment variables and see the logs. I mean, having written all that, I’m wondering why I don’t run it on my Mac instead of Docker Desktop.

Re: Ask HN: How do you manage your “family data warehouse”?

#123

Post by vr46 »

mead5432 wrote: post #73
vr46 wrote: post #24

A Synology NAS running Portainer ( https://www.portainer.io/ ) running Paperless NGX ( https://github.com/paperless-ngx/paperless-ngx ) This works better than I can possibly tell you. I have an Epson WorkForce ES-580W that I bought when my mother passed away to bulk scan documents and it scans everything, double-sided if required, multi-page PDFs if required, at very high speed and uploads everything to OneDrive, at…

I have Paperless watching a network folder from my NAS. The scanner scans directly into that folder and Paperless picks it up and processes it within a couple seconds. Even if Paperless went down for a bit (which happened before I upgraded my server), the files remain on the shared drive until it comes back online. Without that, I had paper piling up waiting to be scanned.

That’s quite cool, I might add that. I did set up E-Mail watching using an iCloud account and it’s working well too. Must go donate some money to the Paperless team!

Re: Ask HN: How do you manage your “family data warehouse”?

#124

Post by BorisMelnik »

owncloud, offline. will back it up once a year or so as well. thought about sending it to Veracrypt and sticking it on a cloud somewhere just in case, but I dont think I have to.

Google Stack (no backups enabled) is great for paper > PDF, then I use a tool for Android called FolderSync Pro to send everything on my phone to a folder on my local network. Once a month or so I organize it. Thought about automating this a bit more but I hate having to spend time while going about my day naming/tagging files.

as far as retaining things, inside each folder I just make an "archive" folder where I stick things that get to be too old. definitely don't overthink the system or it'll turn into a full time job :-)

Re: Ask HN: How do you manage your “family data warehouse”?

#125

Post by crabbone »

coderatlarge wrote: post #115
crabbone wrote: post #32

Having built complicated storage systems during work hours I had a strong urge to build a home NAS. What stops me is that I have this Documents directory on my laptop, which was salvaged from another disk that failed about three years ago, and served for 11 years prior to that, and some documents were copied to it from a PC I owned before then... and going farther back doesn't really make sense as none of the documen…

I've personally been asked by authorities to produce documents that are 30+ years old. It's a bit surreal and I wonder if they could even authenticate them...

I only have a handful of documents that are about that age: my birth certificate and high-school graduation diploma.

Here's an interesting aspect of these: I did lose my college papers a while ago, and when I asked my college if they can produce a copy, they told me that they keep their records for just seven years. So, they had nothing. The high-school I went to doesn't exist anymore. So, if I had to recover my high-school-related papers, I'd have none. But, if I really had to, I could recover my birth certificate, as those don't seem to have an expiration date.

Anyways. In terms of storing digital copies, very little of that is relevant. Even ten years ago most authorities wouldn't be able to accept digital copies. Even today, the municipality of the city I live in won't take digital copies. There's no process through which I could have a digital copy they would recognize.

So, I need to keep hard copies. And the digital copies? -- well, they are perishable. I keep them for convenience, because every now and then I run into some situation where they are accepted. But losing them won't be a big deal as I can always make more of those. So, I don't need any exceptionally reliable storage requirement for the digital documents. It's more like a cache, which can be eventually retired.

Re: Ask HN: How do you manage your “family data warehouse”?

#126

Post by crabbone »

rovr138 wrote: post #43
crabbone wrote: post #32

Having built complicated storage systems during work hours I had a strong urge to build a home NAS. What stops me is that I have this Documents directory on my laptop, which was salvaged from another disk that failed about three years ago, and served for 11 years prior to that, and some documents were copied to it from a PC I owned before then... and going farther back doesn't really make sense as none of the documen…

Any backup? Curious how you do it

I have to keep hard copies anyways. Plenty of places don't take digital copies and have no process through which they would. So, digital documents are only a convenience for the rare situations when someone will take them. And if I lose them -- well, I'll just take another picture.

Re: Ask HN: How do you manage your “family data warehouse”?

#127

Post by jncfhnb »

coderatlarge wrote: post #100
jncfhnb wrote: post #23

Laptop documents folder and outlook inbox

If you're willing to share: curious how you structure your folders (all chronological? by entity producing the document? by your own use?). And also whether you periodically export your outlook inbox and, if so, what you do with the export.

Sort by date modified. A few things stashed in dedicated sub folders like tax returns. No exports.

Re: Ask HN: How do you manage your “family data warehouse”?

#128

Post by newaccount74 »

coderatlarge wrote: post #107
newaccount74 wrote: post #29

We scan every document and put it in a folder synced with SyncThing. We throw the original away unless it is something official like a birth certificate. We use SyncThing because it is free, doesn't require user accounts, and works across Mac, Windows, Linux, iOS (Möbius) and Android. We have a bunch of high level folders (eg. Health, School, etc). Inside that folder every document is named starting with a date, eg:…

thanks for pointing to SyncThing - I'll check it out. if you're willing to share: 1. what scanner do you use? 2. what do you do for documents that might be in two places? ex: car insurance, health insurance - in the insurance folder or the health folder? ex: medications that both spouses take, in his folder or her folder or in a "both" folder?

1) We have a HP all-in-one laser printer/scanner. I wouldn't recommend it, but it's good enough for documents. The one feature I like about it is that it you can scan to a USB stick, which is nice because the scanner software on both Mac and Windows is often unreliable.

2) We have very few categories, so it's usually clear where something goes. If not, I just pick a random folder, usually I search with Spotlight (full text search) anyway.

Re: Ask HN: How do you manage your “family data warehouse”?

#129

Post by newaccount74 »

morisy wrote: post #45

A few folks have talked about digitizing paper documents, so one of my favorite tricks for iPhone havers: iPhones have very nice scanning tools built in, but it’s buried in the Notes app. Create a new note, click the camera icon, then “scan documents” and it will create a very nice, usually well cropped and OCRd scan that’s saved as a PDF you can then export elsewhere. Wish it was a standalone app because it works so…

There is an easier way now!

In the "Files" app, tap the (...) icon in the top right corner and select "Scan Documents".

Re: Ask HN: How do you manage your “family data warehouse”?

#130

Post by KittenPassingBy »

coderatlarge wrote: post #113
KittenPassingBy wrote: post #87

I use paperless-ngx with 3-2-1 backup. It runs OCR and indexes them to makes everything searchable. I can go back years later and search for “toyota” find my car warranty agreement in an instance.

how active is paperless-ngx? it seems very much oriented in the direction I'd like to go. I once imagined setting up a camera rig over my desk so that any paper that I looked at got auto-captured and automatically archived without any explicit effort on my part :)

Paperless-ngx is the current active project. It is very active. It was forked off of paperless-ng when melendez became inactive. Which is also forked off of the original paperless when it became inactive.
Post reply on HN