My Knowledge Lakehouse

This post probably has some kernels of value, but it suffers from the XY problem: it's a note taking system which provides the solution, but not the problem it's solving. Let's invert that and focus on the problem first. Below are the problems my personal system solves, followed by the solutions. My solutions may not fit your situation, but if you see yourself in the problems, maybe you'll be inspired to find the solution that works for you.

1. Formatting easily becomes a distraction, but a little formatting is vital, so I use markdown to write my plans and notes.

2. Mixing streams from different projects is confusing, so each project has its own workspace - either a folder or a distinct prefix on the file name.

3. Work or ideas often become irrelevant for a while as plans change, but I hate feeling like I might lose work that might be valuable later. So I have a "dump" file or folder where I can dump such things. Entries are typically dated so they can be referenced by other files (see below). I typically use a level 1 or 2 markdown section header to delimit entries.

4. I often have a "log" markdown file where I append descriptions of indisputably important project developments (as opposed to maybe-important-later stuff that goes in the dump file). Entries here are dated and delimited similarly to the dump file.

5. There is a "plan" file that I try to keep clean and concise. This is where I go to remember what I have resolved to do next.

6. A common issue with plan files is that they get bloated with discussion of current or past context, as you try to figure out what to do and how to do it. When this happens, the discussion is moved to the "log" or "dump" files and replaced with a "link" describing how to find it again (e.g. "see log date 2024-01-12 for more on this").

7. There is a master plan file that coordinates all the projects. Bloat in this file is moved down to individual project files and replaced with a link.

To recap: markdown; project workspaces; log and dump with dated entries; concise plan file; move bloat out of plan into log or dump and replace with a link; master plan file with similar bloat management strategy.

I have used this system for five years. It has developed over time, but all elements have been in use for at least two years.

I am a staff data scientist leading a team of a half dozen people that is overhauling the core analytics pipelines and models at a Fortune 100, and 80% of my day is IC work. I have this job not because anyone asked me to do it, but because I proved I could do it on my own time. I work normal hours, parent of two kids, working spouse. I don't have time to putter around. This system is a big part of how I make it work.

I just create one markdown file per day. I wrote a custom obsidian plugin that displays the "timeline" as if it was one open file and creates the files in year/month/week subfolders.

But I don't use linking and all the other complexity. But if I ever needed it I could.

Then it's synced with next cloud and also available on my phone.

I like this the most because just as explained, the time line is the most accurate representation of reality. And I can very quickly scroll back to see what I did last week or do full text search.

Simple and useful!

CTRL-F or grep seem like they would work well

I use Apple Notes and don’t bother to organize anything unless it’s a whole bunch of related material that I’m going to use together—notes for an rpg session, say. All the rest, I just make sure that I give it a title that I’ll recognize when I see it, and include words that I’ll likely use if I go looking for it with search.

Been doing this for years, haven’t lost anything yet. It works so well that time organizing it would have been wasted.

Can you explain the timeline further? I don’t understand- just a list of links?