Building personal search infrastructure for your knowledge and code

I've been thinking a lot about how I manage my own data lately (notes, photos, code, reference material, etc) and have concluded that the primary feature I'm looking for is longevity. I'm saddened by the amount of data I've lost over the years, either because of hard disk failures or third-party services going out of business/making it difficult to extract things/getting too expensive.

In light of this, I'm biasing toward simple file formats managed by tools I write myself, and optimizing for cost in a way that I otherwise don't, since any recurring costs incurred by the system are effectively a lifelong commitment. I am relying on S3 for primary storage (so that it is accessible anywhere) but with a sync to offline backup.

So far, I've implemented a personal Zettelkasten tool (with built-in spaced repetition, so doubles as an Anki replacement) and a search engine that's based on Presto (via AWS Athena) so that I don't need to keep an Elasticsearch instance alive. I'm planning to build out other repository tools as I go.

It's been very liberating to build tools that are never meant to be used by anyone other than myself, and with the confidence that the tools don't matter too much anyway since the underlying files are stored in evergreen formats.

Seems similar to ZIM (https://zim-wiki.org/), except proprietary/hosted? I've just started using zim - can someone more experienced compare the two?

I wrote and use daily http://onemodel.org (AGPL, uses postgres), for many reasons listed there :) . One way to think of its current state is a text-mode, easy-to-learn (i hope) infinite mind map of things, where I store and can query effectively everything: calendar, reminders, quasi-anki-like knowledge review, journal, automatic activity log, notes on subjects, very efficiently for the user. (It also stores documents, but that is not very smooth compared to other document systems, nor is browser integration smooth at all.)

Edit: It also has a very basic security model (private, public, unspecified), and with that in mind, can export trees of notes as html or as outline documents (text), with or w/o indentation & numbering, which I've found very useful. And anything can be in as many places in the tree as is helpful. The export to simple html, I use to generate my 2 web sites.

(I plan to move it to Rust, and maybe sqlite, eventually, as well as add features like anki, internal code attached to entity classes for cheap internal customization/automation, etc, but have been slow lately.)

(Edit: it is currently only self-hosted by each user. Have considered doing hosting for other users, and might some day.)

No mentions of https://tiddlywiki.com/?

We've been building something to solve this exact problem. We started looking at how much we could derive from email notifications but parsing unstructured emails is very error-prone and typically incomplete so we decided to lean on external APIs where possible. Essentially what we made was a platform to index your data across services in one searchable document store or build new applications natively on this database where all of your data is one place. If you're interested in getting early access (we're still in early alpha) you can sign up on http://www.aspen.cloud