Live data from Hacker News

Ask HN: How do you manage your “family data warehouse”?

news.ycombinator.com

1–10 of 141 posts

Ask HN: How do you manage your “family data warehouse”?

#1
By "Family Data Warehouse" I refer to all the info and documents that one tracks over the years, perhaps for one's self, but then also potentially for spouses, kids, parents, siblings, relatives, etc.

Some common categories:

(1) Financial: account statements (pdf, paper, etc), transaction CSV/ofx/etc, insurance policies, financial institution correspondence (confirmations, T&Cs, etc), tax forms, deeds/titles, loans, debts, contemporaneous records, paystubs, employment letters, receipts, etc

(2) Health: test results, diagnoses, medications used, doctor info/correspondence, vaccinations, hospital interactions, statements, bills, etc

(3) Product and services info: products purchased, maintenance info, warranties, replacement parts, recall notices, class action events, professional used (CPAs, lawyers, contractors, plumbers, etc), etc

(4) Personal legal and other documents: wills, trusts, health directives, pre-nups, legal settlements, personal contracts, rental agreements, government IDs, immigration/naturalization docs, law enforcement interactions, etc

(5) Memories and other info of sentimental or other value: pictures, videos, cards, recordings, music, diaries, clippings, personal notes, etc

For many of these one might ask the usual questions:

A. do I have a reason/obligation to retain it? for example, tax authorities often require certain classes of record keeping. sometimes commercial entities make false claims in retrospect that you can only rebut with evidence (even major banks!). some documents can end up only being used by heirs at end-of-life (ex: basis information for assets like a home, which can require proof of improvements over the years, depreciation claimed if the home was ever rented, etc)

B. is there a downside to retaining it? clutter, hard/costly to move, identity/other theft risk, physical decay over time (ex: obsolete storage formats (VHS, mini-DV, blueray)), etc

C. will I (or the appropriate person) be able to find it when I need it? physical paper vs electronic mgmt, indexing, filing, passwords, legacy services for transferring access.

D. can I actually retain it? backups, water damage, fire, theft, physical safe management, etc some online services don't provide any/easy export of data, many have inconsistent retention policies (ex utility companies)

Some of the categories above have specialized software and services dedicated to them (ex: photos services, quickbooks, etc) while some are more ad hoc (goog drive, dropbox, local HDD); even specialized services often fall short on key matters (quickbooks doesn't track statements afaik); many specialized systems have various platform limitations (no turbotax on linux, say); many vendors try to play various lock-in tricks; some vendors go out of business or discontinue products over the years; some institutions are very paper oriented so one perhaps scans or tries to limit paper delivery; some services place data in the cloud creating security exposure; etc etc

Re: Ask HN: How do you manage your “family data warehouse”?

#2
Safe deposit box at the bank for the physical records. A labeled manila envelope stored locally for each of us with the typical "identity" documents, SSN card, birth certificate, etc.

Everything is stored locally, unencrypted, and backed up on a local removable HDD, and on BackBlaze. BackBlaze has saved my bacon more than once.

As for physical objects, they're all temporarily mine, until I pass on. They're either tools, consumables, or tied to personal memories. Everything else can go.

The one significant thing to pass on to others is all the data in D:\source\yyyy\yyyymmdd*, which is all of my photos, including family events, short videos, etc. As best I can, all the faces have be tagged with XMP metadata. (Nobody wants to keep an old box of photos without knowing how those people are related)

     Total Files Listed:
           392934 File(s) 725,479,895,968 bytes
           16103 Dir(s)  1,790,571,331,584 bytes free
  
  D:\Source>
I need to discuss with my spouse and child what they want from there.

Re: Ask HN: How do you manage your “family data warehouse”?

#3
Fairly simple here. Everything goes in iCloud with offline backups via Time machine. Documents are scanned and disposed of where possible - collecting paper is a bad bad bad thing. Identity and required paper documents are kept in a locked drawer. Credentials are stored in Keepass. Photos and videos are all in Apple Photos. Instructions are left on how to deal with the offline backup should anyone need it.

Obligation wise it's good to keep all documents for whatever the longest record keeping period is in your region.

Important that you scan your identity documents and keep copies too. That has been incredibly useful when I'm abroad because a lot of people will retain passports etc if you go in a hospital or something so you can just show them the electronic copy and tell them you left it in the hotel.

Re: Ask HN: How do you manage your “family data warehouse”?

#4
Documents are in Subversion hosted on a local home Debian server which is also NAS.

Photos (hundreds of GBs) are also in Subversion hosted on same server. Photos are managed/viewed either via Digikam (on both Mac, Linux, Windows) as well as through a generated HTML/js gallery served on the same server.

Home server is backed up via restic to

* local harddrives that are rotated weekly and moved offsite. * to a rented physical Debian server in a different country * to a cloud storage provider

Server storage is fully encrypted using LUKS. Backups are encrypted via restic/ssh.

Various sensitive documents are separately encrypted inside the svn repos via 'age' or similar tools.

This setup has been working for almost two decades. I have yet to lose any data.

Edit: For documents, we have a personal repo for each family member. Typically, we use a folder structure based on https://johnnydecimal.com/

We also have a repo specifically for receiped in which receipts are just stored as //-.pdf (or .jpg if it is a scan/photo).

Legal documents like wills, etc. are stored in a safe as well as a physical copy stored at a relative + digital versions stored in the svn repo.

Re: Ask HN: How do you manage your “family data warehouse”?

#5
post #4

Documents are in Subversion hosted on a local home Debian server which is also NAS. Photos (hundreds of GBs) are also in Subversion hosted on same server. Photos are managed/viewed either via Digikam (on both Mac, Linux, Windows) as well as through a generated HTML/js gallery served on the same server. Home server is backed up via restic to * local harddrives that are rotated weekly and moved offsite. * to a rented p…

Honest question: why do you use a VCS to store the documents? The kind of documents OP is talking about don't get updates, same for the photos.

Re: Ask HN: How do you manage your “family data warehouse”?

#6
All types of documents go into a paperless ngx. All documents are tagged properly so that I can find them again. Also the search worked quite well so far. I’m still archiving the previous years currently, as I set up everything this year.

Backups via restic end up on backblaze. Haven’t gotten around to set up the local backup. This is a good reminder to do so. :)

The photos end up in iCloud and Adobe with a backup in backblaze. Not so happy yet with this solution. But most alternatives I tried weren’t as comfortable.

Edit: The paperless ngx server is a low power machine that runs in my local network.

Re: Ask HN: How do you manage your “family data warehouse”?

#7
I live in a paperful country, and I have no scanner.

Important stuff has a photo taken, and it's likely backed up in my photo archives (Syncthing → Pi Storage → NAS backup).

My girlfriend still uses google and whatsapp, and I tend to send her important media in our chats, so we'd both have to be kicked out of our gmail and whatsapp accounts to truly lose everything there.

Re: Ask HN: How do you manage your “family data warehouse”?

#8
post #5
post #4

Documents are in Subversion hosted on a local home Debian server which is also NAS. Photos (hundreds of GBs) are also in Subversion hosted on same server. Photos are managed/viewed either via Digikam (on both Mac, Linux, Windows) as well as through a generated HTML/js gallery served on the same server. Home server is backed up via restic to * local harddrives that are rotated weekly and moved offsite. * to a rented p…

Honest question: why do you use a VCS to store the documents? The kind of documents OP is talking about don't get updates, same for the photos.

[deleted]

Re: Ask HN: How do you manage your “family data warehouse”?

#9
post #5
post #4

Documents are in Subversion hosted on a local home Debian server which is also NAS. Photos (hundreds of GBs) are also in Subversion hosted on same server. Photos are managed/viewed either via Digikam (on both Mac, Linux, Windows) as well as through a generated HTML/js gallery served on the same server. Home server is backed up via restic to * local harddrives that are rotated weekly and moved offsite. * to a rented p…

Honest question: why do you use a VCS to store the documents? The kind of documents OP is talking about don't get updates, same for the photos.

Great question. The reason is that my documents do get updated. This is not merely an archive as I also store various more live documents in it like budgets, etc. Some documents are indeed purely archived. But even for those, I find it really handy to have a complete log of when they were added. Who added them, etc. All this is stored in SVN. Also, SVN provides a really easy way to sync the folder to various devices (svn CLI tool on Linux and Mac, and TortoiseSVN on Windows clients).

I am aware that nowadays, SVN is not the modern choice compared to Git. When I designed this system, Git was not available. But I actually find that SVN's centralized architecture fits really well for this purpose.

For photos, I really like having them in SVN as I never lose any image even if I modify it. Sure, it takes up a little extra storage. But storage is cheap. Also, SVN here acts as an extra local backup.

Re: Ask HN: How do you manage your “family data warehouse”?

#10
Dropbox for everything except photos, videos and security-related info.

I used to keep photos and videos on an external hard disk with a jwz-style backup mechanism and Lightroom 3 for organizing it (kept the light room database file on the hard disk itself)

Now just gave up and use iCloud Photos. Not as good as Lightroom but can't find anything better that's also easy.

Secure details go into 1Password with a printout of emergency keys in a wardrobe under lock.

I used to use Evernote but moved away from it soon after they started introducing chat and a bunch of crap no one cared about.

I keep copies of often-referenced items from Dropbox in Apple Notes but the master copy is a file in Dropbox.

I'm using the Shortcuts app on the phone to take a receipt from an online payment and file it away in the /receipts folder in Dropbox.

I used to have an IFTTT integration to take all the photos I posted on Facebook and keep a copy of them in a Dropbox folder as well. It probably still works but I don't really post stuff anywhere any more.

Post reply on HN