Focus on learning only a few things and get really good a them. That reduces the need to organise anything.
Ask HN: How to organize personal knowledge?
211–220 of 228 posts
Re: Ask HN: How to organize personal knowledge?
#212I'm not sure why noone in 150 comments recommended RDBMS yet, but it's great way to store information in general. I put a lot of personal things into RDBMS either imported from external sources, or added manually: invoices, double accounting, time tracking info/worklogs, bank account records, phone company records, passwords, email, irc/chat logs, things to download, things I bought in aliexpress incl. description an…
I'd be interested in hearing more about the system you use. Have you written anything about it before, I'm curious about more specific examples around what you're doing. Do you automate most/some of what you put in? I've thought about writing some scripts to export things from various online accounts/services/apis and importing them into a personal database mostly for backup purposes. It sounds like you're already do…
I don't know where to start, so here are some random thoughts.
- Some of the data import is for archival purposes and mostly one-shot. I'm old enough to have seen many services where I put my data in, or where I had interesting conversations, go and delete everyhing in the process.
- Some of it is for ease of access and offline use. For example Jira can be a very slow system, where each action takes 10 seconds to complete. Some services have too bloated an UI to be useful on slow computers. You can easily lose access to some data/technical comments if permissions change, etc. If you have data locally, it's possible to avoid all those problems and present the data in any way you want.
Yes, I automate imports from external services. In the past I used PHP, now I mostly use Node or Electron, because it has a transparent support for binary data and JSON columns in postgresql, and if I want to parallelize HTTP requests it's easier to do in Node.
Import/sync scripts always have the same structure. First there's some way to gather the data from the external service, then I import it into DB with update/upsert/insert helpers, for example:
upsert(db, 'table_name', {id: 123, name: 'qwe', ...}, ['id']);
One other method I use, with web services that are too messy or complicated, is to create a userscript for Violentmonkey that gathers data as I browse the website and posts it to the database via a small localhost http service. This is useful for JS heavy websites that don't have nice JSON APIs. The script just runs on the background (or can be triggered by a keyboard shortcut) and uses the current state of the DOM to get the data and send them to the database. This method of import is invisible to the service itself.
Most of the scripts are fairly simple (Because I use upsert most of the time, I can and I sometimes do add some extra columns for annotation purposes. For example I mentined Jira. I added columns for ordering issues and for marking them with whether I wait for some feedback or not, and then I have a simple Electron app for ordering and listing my issues that I need to work on where issues that wait for external feedback are not shown until the feedback is provided. It's uncluttered, and as fast as you can imagine, being backed by a localhost database.
In Node I use: https://github.com/request/request-promise-native and https://github.com/request/request and https://github.com/vitaly-t/pg-promise
PostgreSQL replication is well docummented elsewhere. I replicate the entire cluster, which doesn't require any maintenance when creating databases/tables, so it's absolutely painless after the initial setup. All you need to do is check the logs from time to time. All I do is that I run the backup database server on a different UNIX socket and disable the TCP/IP interface. It is very flexible, as you can also put the backup server on a different machine, you can have multiple backup servers, chain them and it just works even if you have intermittent connectivity between the machines.
Most of my system is not universal or systematic and is very specific to my use cases. But that will be true for anyone wishing to preserve some of their data in a world that is fairly hostile to data portability. In fact, lack of universality/configurability reduces the complexity by a lot and makes it all easily manageable for a single person with at most a few hours/data source of time investment.
Re: Ask HN: How to organize personal knowledge?
#213Links: pinboard.in Wiki: I host my own mediawiki instance. Documents: I host my own upspin: upspin.io backed by ZFS. Images: Combination of upspin (above) and iCloud Photos.
I also host my own MediaWiki instance, complete with VisualEditor and a mobile skin. Out of curiosity, why are you not loading documents and images into your wiki? For images, I don't use it as an image repository, just for images of things specific to personal knowledge.
Well in general I want documents to be available locally. If I were to upload them to the wiki, I'd had to 1) download them from somewhere, 2) upload them to the wiki, 3) when I need them, download them again for the wiki. In contrast, if I use upspin, I just download it in a directory served by upspin, so it's just step #1 above, and I always have them available.
Re: Ask HN: How to organize personal knowledge?
#214Earlier quoted context omitted.
Would you upload pdfs and things like that to your mediawiki? Bills, invoices, paychecks, for example.
I do upload pdfs including bills and invoices (not paychecks since I don't have a use case for that). My wiki is private so it's only available to me. Plus I have extensions that allow me to render the pdfs nicely and index all the text in pdfs for easier searching.
I have both a public and a private wiki, but most of my PDFs are 5000+ page instruction set architecture monsters, so I much rather prefer to view them offline in some decent viewer instead of viewing them in the browser.
I also have passports scans and other sensitive stuff. I don't trust that I can secure mediawiki well enough to put those in clear on the internet (in upspin they are encrypted with a local key and the storage is my own server, mediawiki is in the public cloud).
Re: Ask HN: How to organize personal knowledge?
#215>collected a large amount of ebooks, papers, articles, movies, pictures, etc. (all digital). Since you mentioned those specific examples, I see a commonality and so I'm going to assume you're primarily concerned about organizing files that are not authored/generated by you . (Except possibly pictures/photos since you might be talking about jpgs from your personal iPhone or digital camera.) This distinction is key bec…
Apologies for the naive question, but how are the native filesystems for metadata storage, like say if you wanted to store Author for a video, or Location for a photo? Do they have preset available fields, or can they store an arbitrary number of key-value pairs?
Well, there are several tag formats, ie Id3v1, Id3v2, Vorbis comments, APE tags, which have have most needed metadata keys already part of the spec and many can add custom pairs.
Re: Ask HN: How to organize personal knowledge?
#216Earlier quoted context omitted.
While I agree leveraging the filesystem is the way to go in order to sort binary data (ebooks, papers, movies, etc), it's not true org-mode is not suitable to organize those. Since it supports hyperlinks to a wide variety of data, and Emacs is so easy to script, you can definitely do that. For example, org-ref has a lot of functionality implemented to organize academic papers, including the corresponding PDFs. In fac…
How do you mimic the desk feature (lay out a few cars on the table) in org mode?
You can do this in many ways. With dired and several windows, or a view like org agenda that let's you explore several cards concurrently.
Re: Ask HN: How to organize personal knowledge?
#217Earlier quoted context omitted.
Would you upload pdfs and things like that to your mediawiki? Bills, invoices, paychecks, for example.
I do upload pdfs including bills and invoices (not paychecks since I don't have a use case for that). My wiki is private so it's only available to me. Plus I have extensions that allow me to render the pdfs nicely and index all the text in pdfs for easier searching.
What do you use for indexing/searching pdfs? I didn't think about that even being a feature for something like mediawiki, but I guess that would make sense.
One of the big features I like in Evernote is the ability to search pdfs and images. I have a lot of technical manuals that I save, along with receipts/etc that are easier to search for.
Re: Ask HN: How to organize personal knowledge?
#218Earlier quoted context omitted.
I do upload pdfs including bills and invoices (not paychecks since I don't have a use case for that). My wiki is private so it's only available to me. Plus I have extensions that allow me to render the pdfs nicely and index all the text in pdfs for easier searching.
I have both a public and a private wiki, but most of my PDFs are 5000+ page instruction set architecture monsters, so I much rather prefer to view them offline in some decent viewer instead of viewing them in the browser. I also have passports scans and other sensitive stuff. I don't trust that I can secure mediawiki well enough to put those in clear on the internet (in upspin they are encrypted with a local key and…
I've never actually heard of upspin before this thread. Their homepage doesn't give a clear view of what it actually does, but I'm looking over https://upspin.io/doc/overview.md now
Re: Ask HN: How to organize personal knowledge?
#219I'm surprised to not see any YAGNI voices in the comments. I set up Evernote with lots of collected papers I had read. However, I found I would rarely go back, and finding things within a collection of two hundred items was already not easier than just googling it. Perhaps with improved semantic search the situation may improve. I just think summarizing down to a small set of core ideas and storing them in a single s…
Yeah. Summarize and Anki. Otherwise you are just collecting.
Re: Ask HN: How to organize personal knowledge?
#220Earlier quoted context omitted.
I have both a public and a private wiki, but most of my PDFs are 5000+ page instruction set architecture monsters, so I much rather prefer to view them offline in some decent viewer instead of viewing them in the browser. I also have passports scans and other sensitive stuff. I don't trust that I can secure mediawiki well enough to put those in clear on the internet (in upspin they are encrypted with a local key and…
I've never actually heard of upspin before this thread. Their homepage doesn't give a clear view of what it actually does, but I'm looking over https://upspin.io/doc/overview.md now
This video is pretty good: https://www.youtube.com/watch?v=ENLWEfi0Tkg