Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you manage your “family data warehouse”?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 141 posts

Re: Ask HN: How do you manage your “family data warehouse”?

#51

Post by schube »

I scan all documents and paper mail and store the scans in a git annex repo with a few replicas on machines I own or rent. Originals go into binders, ordered by the hash of the document. With a few helper scripts and OCR with Tesseract searching through my documents is surprisingly fast, so is adding new documents at the proper place in the binders. The idea behind ordering by sha1 of the scan was that the binders would be storage only and all categorizing and organizing should be digital, and also that as I add more documents load balancing between the binders is trivial: at the moment I have one binder for hash prefixes 0-7 and one for 8-f, with a divider for each one-hex-digit prefix. If I ever feel there are too many docs per divider I can split further.

Emails go into a plain git repo, and photos go into another git annex repo.

Ideally I’d like to put all of that into Perkeep because I like the idea of just throwing anonymous blobs into a bucket with the possibility to organize and tag them after the fact, but it looks like that project is dead and I would need to add a few things to it to support my use cases.

I store a lot of scans and emails I won’t ever need again. I just consider that storage becomes cheaper faster than my storage needs increase, and that storing extra documents and saving myself the need to decide whether it’s worth keeping, is a good tradeoff.

Re: Ask HN: How do you manage your “family data warehouse”?

#52

Post by EvanAnderson »

hosteur wrote: post #9
GTP wrote: post #5

Earlier quoted context omitted.

Honest question: why do you use a VCS to store the documents? The kind of documents OP is talking about don't get updates, same for the photos.

Great question. The reason is that my documents do get updated. This is not merely an archive as I also store various more live documents in it like budgets, etc. Some documents are indeed purely archived. But even for those, I find it really handy to have a complete log of when they were added. Who added them, etc. All this is stored in SVN. Also, SVN provides a really easy way to sync the folder to various devices…

I've used SVN in a similar capacity for personal and business use, and even at some of my Customers. It's very handy and TortoiseSVN provides a good user experience. I got into doing this back in 2004 and even though Git might be a better tool today (maybe) I've stuck with SVN.

Re: Ask HN: How do you manage your “family data warehouse”?

#53

Post by Helmut10001 »

I separate my data into two categories, fast (and small) and slow (but big). Both types are important, but separation makes management much easier.

Fast data:

    - documents, pdfs, config files, notes txt etc.
    - these files are synced directly, on-site and off-site
Slow data:

    - archives, images, music, large backups etc.
    - these files are transfered to off-site archive on a per-need basis
    - they are made available in the (family) cloud as read-only
    - they are made available to selected internal services read-only (music service, photo service etc.)
Tech stack:

    - I self-host
    - I follow best practices (zfs, 3-2-1 rule, vpn-only)
    - currently I use Nextcloud, rsync and borgmatic

Re: Ask HN: How do you manage your “family data warehouse”?

#54

Post by blakblakarak »

A folder on my desktop named “stuff”.

On a more serious note I use MS OneDrive family account and iCloud - I figure if either of those companies goes under there are bigger things to worry about. I print a lot of my photos so I’ve always got hard copies of the kids when they were growing up - everything else is unimportant or replaceable.

Re: Ask HN: How do you manage your “family data warehouse”?

#55

Post by mark_l_watson »

neilv wrote: post #44
brap wrote: post #18

I have a messy drawer where I dump all of this stuff. It’s basically write-only, once I throw something in there I know I’ll never look for it again. For digital stuff it’s mostly the same idea. If it’s email (which is almost always the case), I archive and forget about it. If it’s something else, I’ll throw it in my virtual “messy drawer” which is just a folder in Google Drive. You are getting quite a few clever ide…

I used to do a box like this. The "temporal ordering" helped narrow search. But I found that finding things in it was still so difficult, that it wasn't worthwhile every time it'd be useful. There was also trying to find one document, and having to do a second thorough pass when I didn't find it in the first pass, which could take over an hour of misery. Today, I routinely download and save PDF/OFX/QFX for every fina…

I use temporal ordering for all paper financial papers I receive. I used to sort into folders by category, but the took too long. At tax time, I do a linear probe for the date range of the tax year.

Since Apple started providing better end to end encryption for Notes and iCloud, my wife and I just use that. We also have a ton of storage on OneDrive, so periodically I ZIP up important stuff, encrypt it and put on OneDrive.

Re: Ask HN: How do you manage your “family data warehouse”?

#56

Post by ludovicianul »

brap wrote: post #18

I have a messy drawer where I dump all of this stuff. It’s basically write-only, once I throw something in there I know I’ll never look for it again. For digital stuff it’s mostly the same idea. If it’s email (which is almost always the case), I archive and forget about it. If it’s something else, I’ll throw it in my virtual “messy drawer” which is just a folder in Google Drive. You are getting quite a few clever ide…

Kind of a similar approach: one drawer for paperwork, google drive for all files, evernote for all notes with a dedicated notebook for different frequent used stuff like IDs, birth certificates, etc. Use search when I need something digital. Periodically throw away warranties and manuals from the big drawer (every 1-2 years) :)

Re: Ask HN: How do you manage your “family data warehouse”?

#57

Post by Brajeshwar »

AlexanderDhoore wrote: post #12

In Belgium, every citizen has access to a "digital vault" provided by the government, which is accessible using their ID card. Individuals can upload up to 1GB of documents into this vault. It's a well-designed system that ensures privacy during one's lifetime, but you can choose what happens with it after your passing. https://www.izimi.be/en/

India has a similar offering called DigiLocker - https://en.wikipedia.org/wiki/DigiLocker

Re: Ask HN: How do you manage your “family data warehouse”?

#58

Post by dudus »

Google photos for photos and videos, I use storage saver which reduces quality but whatever.

Everything else goes into Google drive. I use the Google drive app on Android to scan my physical documents, and I pay for the 200GB plan which is enough for all my stuff.

I've been hoping to write a script to backup stuff into an S3 bucket. I just never have the time.

Re: Ask HN: How do you manage your “family data warehouse”?

#59

Post by tda »

hosteur wrote: post #9
GTP wrote: post #5

Earlier quoted context omitted.

Honest question: why do you use a VCS to store the documents? The kind of documents OP is talking about don't get updates, same for the photos.

Great question. The reason is that my documents do get updated. This is not merely an archive as I also store various more live documents in it like budgets, etc. Some documents are indeed purely archived. But even for those, I find it really handy to have a complete log of when they were added. Who added them, etc. All this is stored in SVN. Also, SVN provides a really easy way to sync the folder to various devices…

At a previous employer, we stored 3Tb of data in SVN. The data was mostly archival, however sometimes corrections were made to historical data. Using partial checkouts it was quite easy to sync a subset locally. I don't use subversion anymore, but thinking about it I prefer the explicit and deliberate commit/checkout/merge over the automagic sync conflict generators that are onedrive, dropbox, nextcloud etc. Also all pf these start to struggle somewhere around 100k files / 1TB data mark.

Re: Ask HN: How do you manage your “family data warehouse”?

#60

Post by neilv »

If you're doing all the data archiving yourself, and only need to make it readable to other family members...

I have a filesystem tree that organizes digital copies of all financial documents/data.

The tree is routinely backed up to rugged USB flash drives, labelmakered "FOR HEIRS", and stored in my safe and bank safe deposit box. The drives are unencrypted, and in FAT32 format (though I use Linux), to increase the likelihood that whomever needs it can access it easily.

At the top of the tree is a one-page `READ-THIS-FIRST.md` (text file using CR-LF encoding, in case they view it on MS Windows). It's also printed as paper copies in the safes. This one-page document tells them about the "FOR HEIRS" USB flash drive, the safe deposit box, GnuCash, and various insurances.

Also in the safes is a one-page printout/PDF from GnuCash, which shows all my assets and liabilities, other than durable goods in the household. (I'm guessing those two one-pagers are most of what heirs would need, at least initially. If they need more later, there's the full GnuCash database, and the neatly named and organized PDF files of things like financial service statements.)

I've recently kludged an easy way to maintain household durable goods inventory, by kludging onto purchase transactions in GnuCash. I still have to find the time to encode all the cruft I've acquired in the past, though. If/once I do, a household inventory might become a third paper copy in the safes (for possible someday insurance or estate purposes).

Post reply on HN