Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Knowledge Lakehouse

tabokie.github.io

Post reply on HN
Print view View original

61–70 of 98 posts

Re: My Knowledge Lakehouse

#61

Post by steve_adams_86 »

rtpg wrote: post #59
dmje wrote: post #51

Obsidian. Total game changer. Obsidian sync is completely worth paying for as well - firstly it helps support the project but also its slick, so well made, and easier than all the hacked together GitHub / s3 / whatever workarounds. From a content point of view I personally find having structure but not too much structure is the way to go. A bit of yaml if needed but just a gentle “daily notes = meeting notes” (on my…

I really wish that Obsidian Sync had like... 5 megs for free or something. I can't justify paying $8/month for a thing I haven't even started using, and without sync I have a hard time saying "OK let's try this" compared to Joplin or Notion. There's a bunch of cool looking stuff though

You can use git and it isn't too difficult or unfriendly. But I totally get it. I'm paying for sync and I do love Obsidian, but it feels a bit steep. I do want them to stick around though, so, I just pay for it.

Re: My Knowledge Lakehouse

#62

Post by LVB »

rtpg wrote: post #59
dmje wrote: post #51

Obsidian. Total game changer. Obsidian sync is completely worth paying for as well - firstly it helps support the project but also its slick, so well made, and easier than all the hacked together GitHub / s3 / whatever workarounds. From a content point of view I personally find having structure but not too much structure is the way to go. A bit of yaml if needed but just a gentle “daily notes = meeting notes” (on my…

I really wish that Obsidian Sync had like... 5 megs for free or something. I can't justify paying $8/month for a thing I haven't even started using, and without sync I have a hard time saying "OK let's try this" compared to Joplin or Notion. There's a bunch of cool looking stuff though

I was in a similar spot. After too much hand wringing I decided to sign up for monthly ($10/month) sync and give it a solid two months. These are important tools that I’m happy to pay for, but they do take some time to evaluate. I figured $20 was a reasonable amount to commit to.

FWIW I ended up going all in on Obsidian and paying yearly, but that outcome was far from a forgone conclusion, even a month in. I went through a full cycle of honeymoon phase, then discovering warts, and then workarounds, etc. It does take time.

Re: My Knowledge Lakehouse

#64

Post by _boffin_ »

threecheese wrote: post #22
_boffin_ wrote: post #5

Thanks for posting this. Currently building out something that is consolidating events from numerous sources and then automatically processing of those. This post helped to give me some additional ideas on the processing of those events.

Interesting, and something I’ve been thinking about. What is your architecture? Using any available workflow products for “retrieve/receive event from X, figure out what do, then do that in Y”?

Right now, since it’s currently being built out for a single user, I’m just using NestJS for the backend, TypeScript and Vite for the browser extension, Python for the daemon, Postgres for the DB and ELK for logging and embeddings storage.

I have a few sources built out right now and some fun utilities thanks to the browser extension, but in the end, I’m expecting to have 15-20 different data sources that are sent by the browser extension, collected by the daemon, and polled via crons in NestJS.

I’m taking ideas i gleaned from the AI-Town paper about taking “memories” and then generating “reflections” and then doing some other stuff on top of that and also building out a knowledge graph of everything i do, read, listen to, etc..

I’m currently figuring out the standardization of the disparate events into memories to then transform into reflections and knowledge graphs.

I’d love to hear more about what you’ve been thinking of and the issues you’ve had.

Edit: One cool thing you can do with the browser extension for this stuff is to send your cookies and headers to the database and then use in the cron jobs to get the data—think pandora.

Re: My Knowledge Lakehouse

#65

Post by orzig »

rtpg wrote: post #59
dmje wrote: post #51

Obsidian. Total game changer. Obsidian sync is completely worth paying for as well - firstly it helps support the project but also its slick, so well made, and easier than all the hacked together GitHub / s3 / whatever workarounds. From a content point of view I personally find having structure but not too much structure is the way to go. A bit of yaml if needed but just a gentle “daily notes = meeting notes” (on my…

I really wish that Obsidian Sync had like... 5 megs for free or something. I can't justify paying $8/month for a thing I haven't even started using, and without sync I have a hard time saying "OK let's try this" compared to Joplin or Notion. There's a bunch of cool looking stuff though

What's your annual salary divided by 2,000 (VERY roughly your hourly rate), and how many months of the service would it buy you as you try it out?

Totally understand if you're not employed, or barely making ends meet, but given the forum I'm going to take a shot in the dark here.

Re: My Knowledge Lakehouse

#66

Post by shrubble »

I have recently started to play around with the lightweight Emacs mode, "HOWM" which was written about 20 years ago by a Japanese author, who still maintains it today.

The howm-mode shows you a summary view, comprised of information (just the title/heading) from the (configurable) last 20 notes you have made. Everything goes under ~/howm/YYYY/MM e.g. ~/howm/2024/01 for notes I am making this month.

It has a simple way to enter schedule, todo, deadline, etc. which it shows just below the header that has single-key commands (s for case-insensitive searching, c to create a new note, etc.) for searching, navigation, new notes.

The genius of howm is that you can write and have very simple back and forward links to any file, URL, or tag, any kind of text etc., and you write "fragments" that is, whatever the smallest unit of text you want - it just needs a title plus anything else you want to add to it.

When you save the file, you hit Ctrl-C plus ",," and are back in the summary view.

So you can have very small sized units/fragments, which are very fast to create and save, but due to the forward and back links and the ease of searching (which will collate in a temporary list, everything found), you can "create fragmentarily but view/search collectively" as the documentation puts it.

My next step is to author HOWM notes in AsciiDoc or Markdown format which can easily be fed (since each note is a separate text file) into a converter for fancier output.

Syncthing easily keeps things in sync between my desktop and laptop.

In the very best of FOSS tradition, someone else wrote up a full PDF manual, in English. https://github.com/Emacs101/howm-manual/blob/main/Howm_tutor...

Re: My Knowledge Lakehouse

#67

Post by BeetleB »

ahmedfromtunis wrote: post #7
dinkleberg wrote: post #3

There are some good ideas in here. Personally I’ve abandoned all of my complex attempts at productivity and knowledge systems and have recently moved to the ever reliable txt file. You can format it however you want, use fun ascii art, or just plain paragraphs. I’ve got a daily.txt where I prepend a new days section every day and put in whatever I want over the course of the day. I also create other files for courses…

It's clearly a nice and simple way to write data, but what about the other way around? How do you manage to find information when you need it?

Although I extensively use org mode, I didn't follow the GP's approach precisely because of this searchability problem. These days, however, vector DBs and doing RAG is getting close to changing my mind. I want to do less and less organizing, and more dumping, and use vector search capabilities to find what I need.

Haven't set it up yet, but it's on my TODO list.

Re: My Knowledge Lakehouse

#69

Post by para_parolu »

rtpg wrote: post #59
dmje wrote: post #51

Obsidian. Total game changer. Obsidian sync is completely worth paying for as well - firstly it helps support the project but also its slick, so well made, and easier than all the hacked together GitHub / s3 / whatever workarounds. From a content point of view I personally find having structure but not too much structure is the way to go. A bit of yaml if needed but just a gentle “daily notes = meeting notes” (on my…

I really wish that Obsidian Sync had like... 5 megs for free or something. I can't justify paying $8/month for a thing I haven't even started using, and without sync I have a hard time saying "OK let's try this" compared to Joplin or Notion. There's a bunch of cool looking stuff though

I come from different perspective: I would love to pay $15 per month if it gives me sync with my private storage and work on ios.

Luckily for both of us there is a solution https://github.com/vrtmrz/obsidian-livesync

Works well on ios.

Re: My Knowledge Lakehouse

#70

Post by accrual »

Several comments about personal note taking systems. Here's mine. :)

    function note { mg +-1 "/home/user/notes/$(date "+%Y-%m-%d").txt"; }
"mg" is "Micro Emacs clone" and "+-1" opens the file at the bottom for appending, but of course it works equally fine with your choice of $EDITOR.

I like this setup because it's plaintext (or markdown if you prefer), easily sorts on filename, and easily greppable. I have a couple years of notes saved this way and it's fun to explore old notes.

Post reply on HN